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Abstract 


The  Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPI®“)  is  designed  to 
provide  benchmark  quality  ratings  relative  to  Capability  Maturity  Model°  Integration 
(CMMI®"^)  models.  It  is  applicable  to  a  wide  range  of  appraisal  usage  modes,  including  both 
internal  process  improvement  and  external  capability  determinations.  SCAMPI  satisfies  all  of 
the  Appraisal  Requirements  for  CMMI  (ARC)  requirements  for  a  Class  A  appraisal  method 
and  can  support  the  conduct  of  ISO/IEC  15504  assessments. 

The  SCAMPI  Method  Definition  Document  describes  the  requirements,  activities,  and  prac¬ 
tices  associated  with  each  of  the  processes  that  compose  the  SCAMPI  method.  It  is  intended 
to  be  one  of  the  elements  of  the  infrastructure  within  which  SCAMPI  Lead  Appraisers  con¬ 
duct  a  SCAMPI  appraisal.  Precise  listings  of  required  practices,  parameters,  and  variation 
limits,  as  well  as  optional  practices  and  guidance  for  enacting  the  method,  are  covered.  An 
overview  of  the  method’s  context,  concepts,  and  architecture  is  also  provided. 


®  Capability  Maturity  Model  is  registered  in  the  U.S.  Patent  and  Trademark  Office. 

™  SCAMPI,  CMMI,  and  CMM  Integration  are  service  marks  of  Carnegie  Mellon  University. 
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About  This  Document 


The  Method  Definition  Document  (MDD)  describes  the  Standard  CMMI^"^  Appraisal  Method 
for  Process  Improvement  (SCAMPI^“).  It  is  intended  to  meet  the  needs  of  different  readers. 
The  document  is  divided  into  three  major  parts,  each  with  a  different  level  of  detail,  intended 
usage,  and  primary  audience.  The  structure,  audiences,  and  suggested  use  of  each  part  of  the 
document  are  described  below. 

Document  Outline 

Part  I:  Overview 

This  part  of  the  document  provides  an  overview  of  the  method’s  context,  concepts,  and  archi¬ 
tecture.  The  reader  is  provided  with  the  big  picture  of  the  method,  rather  than  details  about 
how  to  enact  it.  Table  I-l  shows  the  contents  of  Part  I. 


Table  1-1:  Part  I  Contents 


Section 

Pages 

About  This  Document 

1-3 -1-7 

Executive  Summary 

1-9 -1-13 

SCAMPI  Method  Overview 

1-15  - 1-41 

Part  il:  Process  Definitions 

This  part  of  the  document  provides  the  definitive  coverage  of  method  requirements  and  de¬ 
tailed  activities  and  practices  associated  with  each  of  the  processes  that  compose  the 
SCAMPI  method.  Precise  listings  of  required  practices,  parameters,  and  limits  of  variation 
allowable,  as  well  as  optional  practices  and  guidance  for  enacting  the  method,  are  covered  in 
this  core  part  of  the  document.  Table  1-2  shows  the  contents  of  Part  II. 


CMMI,  CMM  Integration,  and  SCAMPI  are  service  marks  of  Carnegie  Mellon  University. 
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Table  1-2:  Part  II  Contents 


Phase 

Process 

Pages 

1 :  Plan  and  Prepare 
for  Appraisal 

1 . 1  Analyze  Requirements 

II-3  - 11-17 

1.2  Develop  Appraisal  Plan 

11-19-11-31 

1.3  Select  and  Prepare  Team 

11-33  - 11-45 

1.4  Obtain  and  Analyze  Initial  Objective 

Evidence 

11-47  - 11-58 

1.5  Prepare  for  Collection  of  Objective  Evidence 

11-59 -11-69 

2:  Conduct  Appraisal 

2.1  Examine  Objective  Evidence 

11-71  -11-84 

2.2  Verify  and  Validate  Objective  Evidence 

11-85  - 11-97 

2.3  Document  Objective  Evidence 

11-99 -11-109 

2.4  Generate  Appraisal  Results 

II-lll -11-121 

3:  Report  Results 

3. 1  Deliver  Appraisal  Results 

11-123 -11-133 

3.2  Package  and  Archive  Appraisal  Assets 

11-135 -11-143 

Part  III;  Appendices 

The  material  contained  in  the  appendices  of  this  document  provide  further  elaboration  on  se¬ 
lected  topics,  and  are  intended  to  supplement  the  material  in  the  first  two  parts  of  the  docu¬ 
ment.  Rarely  will  someone  who  has  not  already  read  the  first  two  parts  read  an  appendix  of 
this  document.  The  topical  elaboration  and  reference  material  available  in  the  appendices 
help  to  provide  deeper  insight  to  an  already  knowledgeable  reader.  Table  1-3  shows  the  con¬ 
tents  of  Part  III. 


Table  1-3:  Part  III  Contents 


Section 

Pages 

Appendix  A:  Appraisal  Disclosure  Statement 

III-3 

Appendix  B:  The  Role  of  Practice  Implementation  Indicators  in  Verifying 
Practice  Implementation 

III-5-III-11 

Appendix  C:  Focused  Investigation  Elaboration  and  Guidance 

III-13-III-14 

Appendix  D:  ARC/MDD  Traceability  Table 

III-15-III-28 

References 

III-29-in-30 

Glossary 

III-31-in-39 
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Audiences  for  This  Document 

The  MDD  is  primarily  intended  for  SCAMPI  Lead  Appraisers  authorized  by  the  Software 
Engineering  Institute  (SEI).  It  is  expected  that  these  professionals  meet  prerequisites  for 
knowledge  and  skills  specified  by  the  SEI  Appraiser  program  (see  http://www.sei.cmu.edu/ 
for  details),  and  that  this  document  is  one  of  the  elements  of  the  infrastructure  within  which 
they  operate.  They  are  considered  the  primary  audience  for  Part  II.  Candidate  Lead  Apprais¬ 
ers  will  also  use  the  MDD  while  attending  training  to  learn  about  the  method. 

Appraisal  team  members  (under  the  leadership  of  an  authorized  SCAMPI  Lead  Appraiser) 
are  expected  to  refer  to  this  document  as  a  training  aid.  Portions  of  the  document  may  also  be 
used  as  work  aids  during  the  conduct  of  an  appraisal.  Potential  appraisal  team  members  can 
use  the  MDD  to  build  their  knowledge  base  for  future  participation  in  an  appraisal. 


Finally,  the  larger  set  of  stakeholders  for  the  conduct  of  any  given  appraisal  are  also  in  the 
targeted  audience  for  the  document,  particularly  for  Part  I.  These  stakeholders  include: 

•  appraisal  sponsors — leaders  who  sponsor  appraisals  to  meet  business  needs 

•  Process  Group  members — process  improvement  specialists  who  need  to  understand  the 
method,  and  perhaps  help  others  to  gain  familiarity 

•  other  interested  parties  who  wish  to  have  deeper  insight  into  the  methodology  for  pur¬ 
poses  such  as  ensuring  that  they  have  an  informed  basis  for  interpreting  SCAMPI  outputs 
or  making  comparisons  among  similar  methodologies 

How  to  Use  This  Document 
Parti 

It  is  expected  that  every  member  of  the  audience  for  this  document  will  find  value  in  Part  I. 
The  two  primary  sections  in  this  part  are  the  Executive  Summary  and  the  Method  Overview. 

The  Executive  Summary  is  intended  to  provide  high-level  information  about  what  SCAMPI 
is,  and  does  not  require  extensive  knowledge  of  appraisals.  This  portion  of  the  document  may 
be  excerpted  and  provided  to  a  more  casual  reader  or  a  stakeholder  in  need  of  general  infor¬ 
mation  to  support  their  decision  to  conduct  an  appraisal. 


The  Method  Overview  section  provides  more  comprehensive  coverage  of  SCAMPI,  and  can 
be  used  to  begin  building  a  base  of  knowledge  for  readers  who  have  need  of  more  detailed 
information.  Appraisal  sponsors  wanting  more  than  the  summary  view  described  above  will 
want  to  read  this  section.  Every  prospective  SCAMPI  team  leader  and  team  member  is  ex¬ 
pected  to  read  this  section  of  the  document,  to  ensure  that  they  have  the  big  picture  before 
study  of  the  detailed  methodology  begins. 
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Part  II 


People  who  will  enact  an  appraisal  are  expected  to  read  the  second  part  of  the  document. 
Members  of  this  audience  need  to  know  how  to  enact  the  method,  not  just  what  the  method 
is.  Part  II  is  divided  into  Process  Definitions,  which  are  in  turn  divided  into  Activity  Descrip¬ 
tions.  Each  Activity  Description  delineates  Required  Practices,  Parameters  and  Limits,  Op¬ 
tional  Practices,  and  Implementation  Guidance. 

There  are  eleven  processes  contained  in  SCAMPI.  The  processes  (as  defined)  support  a  vari¬ 
ety  of  orderings  and  enactments  to  facilitate  a  variety  of  usage  modes  for  SCAMPI.  The  tem¬ 
poral  flow,  as  well  as  the  flow  of  inputs  and  outputs  among  the  processes,  is  described  in  the 
Method  Overview  section.  The  Process  Definitions  are  not  intended  to  provide  a  start-to- 
finish  view  of  SCAMPI.  Rather,  these  sections  provide  detailed  definitions  of  processes  and 
activities  that  are  invoked  according  to  the  appraisal  plan  created  by  the  appraisal  team 
leader. 

Each  of  the  Process  Definitions  begins  with  a  three-page  overview  of  the  process.  Every 
process  is  defined  by  information  contained  in  the  elements  shown  in  Table  1-4. 


Table  1-4:  Process  Definition  Elements 


Element 

Description 

Purpose 

A  brief  summary  of  what  is  accomplished  by  enacting  the  process 

Entry  Criteria 

Conditions  that  must  be  met  before  enacting  the  process 

Inputs 

Artifacts  or  information  needed  to  enact  the  process 

Activities 

The  set  of  actions  which,  in  combination,  make  up  the  process 

Outputs 

Artifacts  and  assets  that  result  from  enacting  the  process 

Outcome 

Any  change  in  important  conditions  or  artifacts  that  results  from  enact¬ 
ing  the  process 

Exit  Criteria 

Conditions  to  be  met  before  the  process  can  be  considered  complete 

Key  Points 

A  summary  of  the  most  notable  events  associated  with  the  process 

Tools  and 
Techniques 

Work  aids  commonly  used  in  enacting  the  process 

Metrics 

Useful  measures  that  support  the  process  enactment,  or  future  enactments 

Verification  and 
Validation 

Techniques  to  verify  and/or  validate  the  enactment  of  the  process 

Records 

Information  to  be  retained  for  future  use 

Tailoring 

Brief  discussion  of  key  tailoring  options  (not  an  exhaustive  list) 

Interfaces  with 
Other  Processes 

Discussion  of  how  the  process  interacts  with  other  processes  in  the 
method 

Summary  of 
Activities 

A  narrative  summary  of  the  set  of  activities 

1-6 


CMU/SEI-2001-HB-001 


Following  the  three  pages  of  introductory  material,  each  activity  that  is  a  part  of  the  process 
is  briefly  summarized  to  orient  the  reader  to  the  scope  of  the  activity.  Each  Activity  Descrip¬ 
tion  includes  the  elements  shown  in  Table  1-5. 


Table  1-5:  Activity  Description  Elements 


Element 

Description 

Required  Practices 

A  listing  of  practices  that  must  be  implemented  to 
consider  the  enactment  a  valid  SCAMPI 

Parameters  and  Limits 

I 

Acceptable  limits  for  things  that  are  allowed  to  vary, 
and  acceptable  limits  for  things  under  the  discretion  of 
the  appraisal  team  leader 

Optional  Practices 

Actions  that  reflect  good  practice  but  are  not  required 

Implementation  Guidance 

Narrative  description  of  advice  or  things  to  consider  in 
performing  the  activity 

Complete  and  unambiguous  descriptions  of  the  method  processes  and  activities  are  provided 
in  this  part  of  the  document.  In  combination  with  the  training  materials  and  work  aids  that 
compose  the  CMMI  Steward’s  appraiser  program,  this  information  provides  a  firm  basis  for 
standardization  (within  reasonable  limits)  of  the  practice  of  Process  Appraisals. 

Part  III 

The  appendices  of  the  document  provide  detailed  coverage  of  special  topics  as  well  as  refer¬ 
ence  material.  Readers  knowledgeable  in  SCAMPI  are  expected  to  read  these  sections  for 
further  understanding. 

Feedback  Information 

We  are  very  interested  in  your  ideas  for  improving  this  document.  See  the  CMMI  Web  site 
for  information  on  how  to  provide  feedback:  <http://www.sei.cmu.edu/cmmi/products/ 
change-requests.html>. 


If  you  have  questions,  send  an  email  to  cmmi-comments@sei.cmu.edu. 
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Executive  Summary 


What  is  SCAMPI? 

The  Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPI)  is  designed  to 
provide  benchmark  quality  ratings  relative  to  Capability  Maturity  Model  Integration 
(CMMI^*^)  models.  It  is  applicable  to  a  wide  range  of  appraisal  usage  modes,  including  both 
internal  process  improvement  and  external  capability  determinations.  SCAMPI  satisfies  all  of 
the  Appraisal  Requirements  for  CMMI  (ARC)  requirements  for  a  Class  A  appraisal  method 
and  can  support  the  conduct  of  ISO/IEC  15504  assessments. 


SCAMPI  VI.  1  enables  a  sponsor  to 

•  gain  insight  into  an  organization’s  engineering  capability  by  identifying  the  strengths  and 
weaknesses  of  its  current  processes 

•  relate  these  strengths  and  weaknesses  to  the  CMMI  model 

•  prioritize  improvement  plans 

•  focus  on  improvements  (correct  weaknesses  that  generate  risks)  that  are  most  beneficial 
to  the  organization  given  its  current  level  of  organizational  maturity  or  process  capabili¬ 
ties 

•  derive  capability  level  ratings  as  well  as  a  maturity  level  rating 

•  identify  development/acquisition  risks  relative  to  capability/maturity  determinations 

As  a  Class  A  appraisal  method,  SCAMPI  is  an  appropriate  tool  for  benchmarking.  Sponsors 
who  want  to  compare  an  organization’s  process  improvement  achievements  with  other  or¬ 
ganizations  in  the  industry  may  have  a  maturity  level  determined  as  part  of  the  appraisal 
process.  Decisions  made  on  the  basis  of  maturity  level  ratings  are  only  valid  if  the  ratings  are 
based  on  known  criteria.  Consequently,  contextual  information — organizational  scope, 

CMMI  model  scope,  appraisal  method  type,  the  identity  of  the  Lead  Appraiser  and  the 
team — are  items  for  which  criteria  and  guidance  are  provided  within  the  method  to  ensure  a 
consistent  interpretation  within  the  community.  Benchmarking  can  only  be  valid  when  there 
is  a  consistent  basis  for  establishing  the  benchmarks.  The  SEI  maintains  industry  aggregates 
for  appraisal  results.  These  data  are  reported  in  industry  maturity  profiles  gathered  from  or¬ 
ganizations  that  have  performed  appraisals  since  1987.  The  profile  is  based  upon  appraisal 
data  provided  by  SEI-trained  professionals,  and  is  updated  twice  annually. 


CMMI  is  a  service  mark  of  Carnegie  Mellon  University. 
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As  the  CMMI  Steward  ,  the  SEI  supports  the  SCAMPI  method  and  operates  an  authorization 
program  for  SCAMPI  Lead  Appraisers.  Additional  details  can  be  found  at 
http://www.sei.cmu.edu. 

Core  Concepts  and  Approach 

SCAMPI  VI.  1,  as  a  benchmarking  appraisal  method,  relies  upon  an  aggregation  of  evidence 
that  is  collected  via  instruments,  presentations,  documents,  and  interviews.  These  four 
sources  of  data  feed  an  “information-processing  engine”  whose  parts  are  made  up  of  a  series 
of  data  transformations.  The  appraisal  team  observes,  hears,  and  reads  information  that  is 
transformed  first  into  notes,  and  then  into  statements  of  practice  implementation  gaps  or 
strengths  (where  appropriate),  and  then  into  preliminary  findings.  These  are  validated  by  the 
organizational  unit  before  they  become  final  findings.  The  critical  concept  is  that  these  trans¬ 
formations  are  applied  to  data  reflecting  the  enacted  processes  in  the  organizational  unit  and 
the  CMMI  reference  model,  and  this  forms  the  basis  for  ratings  and  other  appraisal  results. 

Planning  is  absolutely  critical  to  the  execution  of  SCAMPI  V 1 . 1 .  All  phase  and  process  ac¬ 
tivities  briefly  discussed  below  derive  from  a  well-articulated  plan  developed  by  the  qualified 
Lead  Appraiser,  in  concert  with  members  of  the  appraised  organization  and  the  appraisal 
sponsor. 

SCAMPI  V1.1  Methodology 

SCAMPI  consists  of  three  phases  and  eleven  essential  processes,  as  was  shown  in  Table  1-2. 
Each  phase  is  described  in  detail  below. 

Phase  1 :  Plan  and  Prepare  for  Appraisal 

The  sponsor’s  objectives  for  performing  SCAMPI  are  determined  in  phase  1,  process  1.1, 
Analyze  Requirements.  All  other  planning,  preparation,  execution,  and  reporting  of  results 
proceed  from  this  initial  activity  according  to  the  phase  and  processes  outlined.  Because  of 
the  significant  investment  and  logistical  planning  involved,  considerable  iteration  and  re¬ 
finement  of  planning  activities  should  be  expected  in  phase  1 .  With  each  subsequent  phase, 
the  amount  of  iteration  will  decrease  as  data  are  collected,  analyzed,  refined,  and  translated 
into  findings  of  significance  relative  to  the  model. 

A  team  of  experienced  and  trained  personnel  performs  a  SCAMPI  over  a  period  of  time  nego¬ 
tiated  by  the  sponsor  and  the  team  leader.  The  scope  of  the  organization  to  be  appraised  (ac¬ 
tual  projects  or  programs  that  will  participate),  as  well  as  the  scope  within  the  CMMI  model 
(process  areas),  must  be  defined  and  agreed  to.  The  scope  of  the  organization  and  model  pro¬ 
vides  the  basis  upon  which  to  estimate  personnel  time  commitments,  logistical  costs  (e.g., 
travel),  and  overall  costs  to  the  appraised  organization  and  to  the  sponsoring  organization. 
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Before  the  appraisal  begins,  members  of  the  appraised  organization  typically  prepare  objec¬ 
tive  evidence.  Objective  evidence  consists  of  qualitative  or  quantitative  information,  records, 
or  statements  of  fact  pertaining  to  the  characteristics  of  an  item  or  service  or  to  the  existence 
and  implementation  of  a  process  element.  It  is  based  on  observation,  measurement,  or  test, 
and  can  be  verified.  During  an  on-site  period,  the  appraisal  team  verifies  and  validates  the 
objective  evidence  provided  by  the  appraised  organization  to  identify  strengths  and  weak¬ 
nesses  relative  to  the  CMMI  reference  model.  The  information-processing  engine  is  thus  fu¬ 
eled  by  the  objective  evidence  already  available,  saving  the  appraisal  team  the  time  and  effort 
of  a  “discovery”  process. 

This  preparation  by  the  appraised  organization  is  critical  to  the  efficient  execution  of  a 
SCAMPI  appraisal.  Analysis  of  preliminary  objective  evidence  provided  by  the  appraised 
organization  plays  a  critical  role  in  setting  the  stage  for  the  appraisal  execution.  If  substantial 
data  are  missing  at  this  point,  subsequent  appraisal  activities  can  be  delayed  or  even  cancelled 
if  the  judgment  is  made  that  continuing  appraisal  activities  will  not  be  sufficient  to  make  up 
for  the  deficiency. 

The  collection  of  objective  evidence  by  the  appraised  organization  in  advance  of  the  appraisal 
not  only  improves  appraisal  team  efficiency,  but  also  offers  several  other  benefits  to  the  or¬ 
ganization: 

•  improved  accuracy  in  appraisal  results  delivered  by  external  appraisal  teams  (i.e.,  clear 
understanding  of  implemented  processes,  strengths,  and  weaknesses) 

•  detailed  understanding  of  how  each  process  instance  has  implemented  appraisal  reference 
model  practices,  and  the  degree  of  compliance  and  tailoring  of  organizational  standard 
processes 

•  facilities  for  monitoring  process  compliance  and  process  improvement  progress 

•  residual  appraisal  assets  that  can  be  reused  on  subsequent  appraisals,  minimizing  the  ef¬ 
fort  necessary  for  preparation 

Phase  2:  Conduct  Appraisal 

In  phase  2,  the  appraisal  team  focuses  on  collecting  data  from  the  appraised  organization  to 
judge  the  extent  to  which  the  model  is  implemented.  Integral  to  this  approach  is  the  concept 
of  “coverage,”  which  implies  (a)  the  collection  of  sufficient  data  for  each  model  component 
within  the  CMMI  reference  model  scope  selected  by  the  sponsor,  and  (b)  obtaining  a  repre¬ 
sentative  sample  of  ongoing  processes  (spanning  the  life-cycle  phases  that  the  appraised  or¬ 
ganization  is  using  in  the  development  and  delivery  of  its  products  and  services).  For  a 
benchmarking  appraisal  methodology,  this  means  collecting  data  and  information  on  all  the 
reference  model  practices  for  each  process  instantiation  being  appraised  within  the  organiza¬ 
tional  unit.  The  data  collection  plan  developed  in  phase  1  undergoes  continuous  iteration  and 
refinement  until  sufficient  coverage  is  achieved. 
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Upon  determining  that  sufficient  coverage  of  the  reference  model  and  organizational  unit  has 
been  obtained,  appraisal  findings  and  ratings  may  be  generated.  Goal  ratings  are  determined 
within  each  process  area,  which  collectively  can  be  used  to  determine  a  capability  level  rating 
for  the  individual  process  areas,  as  well  as  a  process  maturity  rating  for  the  organizational  . 
unit. 

Phase  III:  Report  Results 

In  phase  III,  the  appraisal  team  provides  the  findings  and  ratings  as  appraisal  results  to  the 
appraisal  sponsor  and  the  organization.  These  appraisal  results  become  part  of  the  appraisal 
record,  which  becomes  protected  data  according  to  the  desires  of  the  sponsoring  organization 
and  the  appraised  organization.  The  level  of  protection  and  the  plan  for  the  disposition  of  ap¬ 
praisal  materials  and  data  is  determined  in  phase  I  in  collaboration  with  the  sponsor.  The 
agreed-to  appraisal  record  is  also  forwarded  to  the  CMMI  Steward.  The  Steward  adds  it  to  a 
confidential  database  for  summarization  into  overall  community  maturity  and  capability  level 
profiles,  which  are  made  available  to  the  community  on  a  semiannual  basis. 

SCAMPI  Tailoring 

Successful  application  of  SCAMPI  VI.  1  relies  upon  adjusting  the  parameters  of  the  method 
to  the  needs  of  the  organization  and  to  the  objectives  and  constraints  of  the  sponsor’s  organi¬ 
zation. 


The  sponsor’s  objectives  largely  influence  tailoring  decisions.  The  CMMI  model  scope  and 
representation  (staged  or  continuous),  the  size  of  the  organizational  unit,  the  number  and  size 
of  sampled  projects,  the  size  of  the  appraisal  team,  and  the  number  of  interviews  greatly  in¬ 
fluence  things  such  as  preparation  time,  time  on  site,  and  monetary  costs,  and  so  are  also  ma¬ 
jor  factors  when  choosing  tailoring  options.  All  tailoring  decisions  must  be  documented  in  the 
appraisal  plan. 


Tailoring  should  not  exceed  the  acceptable  limits  allowed  by  the  appraisal  method.  The 
SCAMPI  Lead  Appraiser  is  responsible  for  ensuring  that  the  requirements  of  the  method  are 
satisfied.  Tailoring  the  method  too  severely  could  result  in  the  failure  to  satisfy  method  re¬ 
quirements,  the  inability  to  obtain  sufficient  data  for  generation  of  appraisal  findings  or  rat¬ 
ings,  or  the  failure  to  meet  the  criteria  necessary  for  recognition  as  a  SCAMPI  (ARC  Class  A) 
appraisal. 


Time  Frame  and  Personnel  Requirements 

A  nominal  time  frame  for  conducting  a  SCAMPI  appraisal  is  3  months,  including  planning, 
preparation,  and  execution.  The  follow-on  activities  implicit  with  a  full  cycle  of  appraisal  to 
re-appraisal  would  include  time  for  creating  an  action  plan  and  18  to  24  months  for  imple¬ 
mentation,  with  a  re-appraisal  occurring  in  the  latter  6  months  of  that  period.  (The  time  esti¬ 
mates  given  here  refer  to  calendar  duration  rather  than  person-months  of  effort.) 
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Personnel  needed  to  participate  in  activities  or  perform  tasks  in  a  SCAMPI  appraisal  include 
the  sponsor,  the  appraisal  team  leader,  the  Organizational  Unit  Coordinator  (OUC),  the  se¬ 
lected  participants,  and  appraisal  team  members.  Their  time  commitments  will  vary  a  great 
deal  depending  on  the  specific  parameters  of  the  appraisal  (e.g.,  organizational  scope)  and 
their  role.  Typically,  appraisal  participants  can  expect  to  spend  one  to  three  hours  each  to 
provide  objective  evidence  to  the  team  and  attend  validation  sessions.  On  the  other  extreme, 
the  OUC  may  spend  as  much  as  three  weeks  of  full  time  effort  helping  the  team  and  the  or¬ 
ganization  to  prepare  for  and  conduct  the  appraisal.  Experienced  Lead  Appraisers  will  pro¬ 
vide  effort  estimates  corresponding  to  the  set  of  tailoring  options  they  prefer  to  use  in  con¬ 
ducting  a  SCAMPI  appraisal. 
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SCAMPI  Method  Overview 


This  section  provides  an  overview  of  the  underlying  principles  and  concepts  of  the  SCAMPI 
method.  Readers  of  the  SCAMPI  MDD  should  become  familiar  with  this  material  prior  to 
reading  the  process  descriptions  in  Part  II  of  this  document,  where  the  method  requirements 
and  tailoring  options  are  defined.  This  overview  is  primarily  targeted  at  appraisal  team  lead¬ 
ers  and  appraisal  team  members  who  will  be  performing  SCAMPI  appraisals.  Additional  au¬ 
diences  might  include  appraisal  sponsors  or  process  improvement  professionals  interested  in 
understanding  SCAMPI  features  and  the  results  that  can  be  expected. 

Method  Context 

The  SCAMPI  appraisal  method  is  used  to  identify  strengths,  weaknesses,  and  ratings  relative 
to  CMMI  reference  models.  It  incorporates  best  practices  found  successful  in  the  appraisal 
community,  and  is  based  on  the  features  of  several  legacy  appraisal  methods,  including 

•  CMM-Based  Appraisal  for  Internal  Process  Improvement  (CBA IPI)  V 1 . 1  [Dunaway 
96b]. 

•  Electronic  Industries  Alliance/Interim  Standard  (EIA/IS)  73 1 .2  Appraisal  Method  [EIA 
98b]. 

•  Software  Capability  Evaluation  (SCE)  V3.0  Method  Description  [Byrnes  96] 

•  Software  Development  Capability  Evaluation  (SDCE)  [AFMC  94] 

•  FAA  Appraisal  Method  (FAM)  [Ibrahim  99] 

SCAMPI  satisfies  the  Appraisal  Requirements  for  CMMI  (ARC)  VI.  1  [SEI  Ola]  and  is  a 
Class  A  appraisal  method. 

Method  Objectives  and  Characteristics 

The  SCAMPI  method  has  the  following  primary  objectives: 

•  Provide  a  common,  integrated  appraisal  method  capable  of  supporting  appraisals  in  the 
context  of  internal  process  improvement,  supplier  selection,  and  process  monitoring  (see 
“Modes  of  Usage”). 

•  Provide  an  efficient  appraisal  method  capable  of  being  implemented  within  reasonable 
performance  constraints  (see  “Method  Performance”). 
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The  SCAMPI  method  is  also  designed  to  prioritize  and  satisfy  certain  essential  characteris¬ 
tics,  which  were  obtained  via  community  feedback  and  are  summarized  in  Table  1-6.  These 
have  been  used  as  the  rationale  for  key  method  architecture  and  design  decisions,  which  are 
described  in  this  overview  and  throughout  the  MDD. 


Table  1-6:  Essential  Characteristics  of  the  SCAMPI  Method 


Characteristic 

Explanation 

Accuracy 

Ratings  are  truly  reflective  of  the  organization’s  maturity /capability, 
reflect  the  reference  model,  and  can  be  used  for  comparison  across  or¬ 
ganizations. 

Appraisal  results  reflect  the  strengths  and  weaknesses  of  the  appraised 
organization  (i.e.,  no  significant  strengths  and  weaknesses  are  left  un¬ 
discovered). 

Repeatability 

Ratings  and  findings  of  an  appraisal  are  likely  to  be  consistent  with 
those  of  another  independent  appraisal  conducted  under  comparable 
conditions  (i.e.,  another  appraisal  of  identical  scope  will  produce  consis¬ 
tent  results). 

Cost/Resource 

Effectiveness 

The  appraisal  method  is  efficient  in  terms  of  person-hours  spent  plan¬ 
ning,  preparing,  and  executing  an  appraisal. 

The  method  takes  account  of  the  organizational  investment  in  obtaining 
the  appraisal  results,  including  the  resources  of  the  host  organization, 
the  impact  on  appraised  projects,  and  the  appraisal  team. 

Meaningfulness 
of  Results 

Appraisal  results  are  useful  to  the  appraisal  sponsor  in  supporting  deci¬ 
sion  making.  This  may  include  application  of  the  appraisal  results  in  the 
context  of  internal  process  improvement,  supplier  selection,  or  process 
monitoring. 

ARC  Compliance 

SCAMPI  is  a  Class  A  method  and  complies  with  all  ARC  requirements. 

Modes  of  Usage 

As  used  in  the  CMMI  Product  Suite  materials,  an  appraisal  is  an  examination  of  one  or  more 
processes  by  a  trained  team  of  professionals  using  an  appraisal  reference  model  as  the  basis 
for  determining  strengths  and  weaknesses.  An  appraisal  is  typically  conducted  in  the  context 
of  process  improvement  or  capability  evaluation.  The  term  “appraisal”  is  a  generic  term  used 
throughout  the  CMMI  Product  Suite  to  describe  applications  in  these  contexts,  traditionally 
known  as  assessments  and  evaluations. 


The  basic  difference  between  an  assessment  and  an  evaluation  is  that  an  assessment  is  an  ap¬ 
praisal  that  an  organization  does  to  and  for  itself  for  the  purposes  of  process  improvement. 
Assessments  provide  internal  motivation  for  organizations  to  initiate  or  continue  process  im¬ 
provement  programs.  An  evaluation  is  an  appraisal  in  which  an  external  group  comes  into  an 
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organization  and  examines  its  processes  as  input  to  a  decision  regarding  future  business. 
Evaluations  are  typically  externally  imposed  motivation  for  organizations  to  undertake  proc¬ 
ess  improvement. 

Historically,  assessments  and  evaluations  have  been  performed  using  separate,  but  similar, 
method  descriptions,  training,  infrastructure,  and  assets.  With  version  1.1  of  the  CMMl  Prod¬ 
uct  Suite,  these  are  now  combined  into  a  single,  integrated  appraisal  methodology.  Apart 
from  this  section  of  the  MDD,  the  terms  assessment  and  evaluation  are  not  used;  the  more 
general  term  “appraisal”  is  used  throughout  to  encourage  the  recognition  of  a  single  inte¬ 
grated  method.  Concepts  from  legacy  assessment  and  evaluation  methods  are  borrowed  from 
liberally  in  the  SCAMPI  MDD,  and  representative  experts  in  these  methods  contributed  as 
core  members  of  the  ARC/MDD  product  development  team.  It  is  expected  that  users  familiar 
with  one  or  more  of  those  legacy  methods  will  be  able  to  identify  features  that  will  help  ease 
their  transition  to  the  SCAMPI  method. 


As  an  ARC  Class  A  method,  SCAMPI  is  a  benchmarking-oriented  method  suitable  for  gener¬ 
ating  ratings.  SCAMPI  appraisals  can  be  performed  in  three  modes  of  usage,  as  depicted  in 
Table  1-7.  While  many  of  the  SCAMPI  features  are  common  across  all  usage  modes  (e.g., 
identification  of  strengths,  weaknesses,  and  ratings),  there  are  differences  in  motivation  and 
intent  that  can  result  in  some  expected  method  differences  in  these  usage  modes.  The  method 
may  be  tailored  significantly  to  meet  the  business  objectives  of  the  appraisal  sponsor. 


Table  1-7:  SCAMPI  Modes  of  Usage 


Usage  Mode 

Description 

Internal  Process 
Improvement 

Organizations  use  appraisals  to  appraise  internal  processes,  generally 
to  either  baseline  their  capability/maturity  level(s),  to  establish  or  up¬ 
date  a  process  improvement  program,  or  to  measure  progress  in  im¬ 
plementing  such  a  program.  Applications  include  measuring  process 
improvement  progress,  conducting  process  audits,  focusing  on  specific 
domains  or  product  lines,  appraising  specific  projects,  and  preparing 
for  external  customer-led  appraisals.  In  this  manner,  SCAMPI  apprais¬ 
als  supplement  other  tools  for  implementing  process  improvement  ac¬ 
tivities. 

Supplier  Selection 

Appraisal  results  are  used  as  a  high-value  discriminator  to  select  sup¬ 
pliers.  The  results  are  used  in  characterizing  the  process-related  risk  of 
awarding  a  contract  to  a  supplier.  The  appraisal  results  are  typically 
only  one  criterion  among  many  used  to  select  suppliers.  Results  are 
often  used  as  a  baseline  in  subsequent  process  monitoring  with  the  se¬ 
lected  supplier. 
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Usage  Mode 

Description 

Process  Monitoring 

Appraisal  methods  are  also  used  in  monitoring  processes  (for  example, 
after  contract  award,  by  serving  as  input  for  an  incentive/award  fee 
decision  or  a  risk  management  plan).  The  appraisal  results  are  used  to 
help  the  sponsoring  organization  tailor  its  contract  or  process  monitor¬ 
ing  efforts  by  allowing  it  to  prioritize  efforts  based  on  the  observed 
strengths  and  weaknesses  of  the  supplying  organization’s  processes. 

This  usage  mode  focuses  on  a  long-term  teaming  relationship  between 
the  sponsoring  organization  and  the  development  organization  (buyer 
and  supplier). 

Where  appropriate,  differences  in  the  method  requirements,  tailoring,  or  recommended  im¬ 
plementation  applicable  to  these  usage  modes  are  discussed  in  process  descriptions  and  ac¬ 
tivities  provided  in  Part  11.  These  differences  occur  most  significantly  in  the  planning  proc¬ 
esses  (e.g.,  appraisal  objectives,  sponsorship,  appraisal  planning,  selection  of  participants, 
preparation)  and  reporting  processes  (e.g.,  reporting  of  appraisal  results,  use  of  appraisal  re¬ 
sults  for  decision-making,  and  follow-on  activities). 


Note  that  the  SCAMPI  method  boundary  is  expressed  in  terms  of  enactment  of  the  appraisal 
method,  including  reporting  of  appraisal  results,  but  does  not  address  the  usage  of  those  re¬ 
sults  in  the  sponsor’s  business  context.  For  example,  the  use  of  appraisal  results  to  identify 
acquisition  risks  for  a  supplier  source  selection  is  beyond  the  scope  of  the  method.  These 
concepts  are  better  described  in  other  documentation  specific  to  those  business  contexts,  such 
as  acquisition  regulations,  standards,  and  processes. 

Method  Performance 

Performing  appraisals  efficiently  involves  minimizing  the  use  of  resources  and  the  impact  on 
appraisal  teams  and  appraised  organizations,  while  maintaining  the  essential  method  charac¬ 
teristics  that  ensure  the  high  degree  of  accuracy  required  for  an  effective  benchmarking  ap¬ 
praisal  method.  The  significantly  larger  size  of  the  CMMI  models  relative  to  legacy  source 
models  makes  this  an  even  greater  challenge. 


Method  performance  during  the  on-site  period  was  an  influential  design  driver  that  directly 
resulted  in  many  of  the  SCAMPI  features.  The  MDD  contains  many  recommendations  on 
proven,  effective  practices  that  contribute  positively  to  efficient  appraisals,  although  many  of 
these  may  not  be  strict  requirements  of  the  method.  However,  the  appraisal  method  is  only 
part  of  the  solution  for  efficient  and  cost-effective  benchmarking  appraisals  capable  of  satis¬ 
fying  all  appraisal  objectives.  Appraisal  efficiency  must  also  be  a  commitment  shared  among 
appraisal  sponsors,  appraised  organizations,  and  appraisal  teams. 


Several  means  were  used  to  identify  targets  for  appraisal  performance  improvement,  includ¬ 
ing  analysis  of  CBA IPI  results  [Dunaway  00],  efficiency  features  from  other  appraisal  meth- 
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ods,  CMMI  pilot  appraisals,  and  improvement  suggestions  and  best  practices  solicited  from 
the  Lead  Appraiser  community.  Ideas  were  grouped,  analyzed,  and  prioritized  for  potential 
benefit  to  method  performance  improvement.  Among  the  areas  identified  as  potential  targets 
for  performance  improvement  were  the  following: 


Table  1-8:  Potential  Targets  for  Appraisal  Performance  Improvement 


Performance 
Improvement  Topic 

Summary  Description  and  Examples 

Prework 

Thorough  appraisal  planning.  Greater  organizational  readiness. 
Pre-on-site  data  review. 

Focused  Investigation 

Focus  the  scope  of  investigation  and  follow-up  based  on  documen¬ 
tation  review  and  questionnaire  analysis.  Continually  consolidate 
data  to  determine  progress  toward  sufficient  coverage.  Target  fur¬ 
ther  investigation  and  interviews  on  specific  data  collection  needs 
to  optimize  effort  where  it  is  needed. 

Reuse 

Validate  results  of  prior  appraisals.  Reduce  discovery  of  earlier, 
proven  findings. 

Observations 

Reduce  time  spent  crafting  observations. 

Tailoring 

Provide  greater  clarity  on  mandatory,  suggested,  and  optional  fea¬ 
tures  of  the  method. 

Rating 

Rate  practices  (e.g.,  implemented,  partial,  not  implemented). 

Tools 

Tool  support  is  crucial  for  efficient  data  collection,  analysis,  and 
consolidation. 

Training 

Just-in-time  training.  Use  ‘live  data”  and  tools  for  exercises. 

Assets 

“Look-fors,”  templates,  checklists. 

Several  of  these  performance  improvement  topics  have  been  incorporated  into  the  SCAMPI 
MDD,  either  as  fundamental  method  concepts  (described  in  “Method  Concepts”  below),  or  as 
recommendations  and  implementation  options. 

Since  SCAMPI  is  suitable  for  benchmarking,  thus  requiring  high  confidence  in  ratings,  thor¬ 
oughness  is  necessary.  Organizations  for  which  (a)  generation  of  ratings  is  not  required,  (b) 
the  primary  application  is  identification  of  strengths  and  weaknesses  for  process  improve¬ 
ment,  and  (c)  efficiency  of  appraisal  resources  is  a  primary  concern  may  be  well  advised  to 
consider  alternative  appraisal  approaches.  Their  needs  may  be  satisfied  by  less  demanding 
ARC  Class  B  or  Class  C  methods.  This  is  particularly  true  for  organizations  that  are  early  in 
their  process  improvement  cycle.  Refer  to  “Requirements  for  CMMI  Appraisal  Method  Class 
Structure”  and  “Requirements  for  CMMI  Appraisal  Methods”  in  the  ARC  [SEI  Ola]  for  fur¬ 
ther  discussion  of  these  issues  and  for  guidance  in  selecting  an  appropriate  appraisal  method 
to  fit  your  business  objectives. 
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Method  Concepts 

This  section  provides  a  description  of  fundamental  concepts  employed  by  the  SCAMPI 
method.  These  concepts  are  treated  here  to  provide  readers  with  an  overall  understanding  of 
the  method  prior  to  reading  the  method  Process  Definitions  in  Part  11.  Many  of  these  concepts 
are  distributed  across  several  appraisal  method  processes  or  activities,  so  it  is  important  to 
ensure  that  a  common  understanding  is  obtained  to  recognize  the  components  of  these  con¬ 
cepts  as  they  appear  elsewhere  in  this  document. 

In  addition  to  requirements  of  the  ARC,  these  method  concepts  are  derived  from,  and  heavily 
influenced  by,  the  method  objectives,  essential  method  characteristics,  appraisal  modes  of 
usage,  and  performance  objectives  described  above. 

Method  Assumptions  and  Design  Principles 

In  addition  to  the  factors  just  mentioned,  SCAMPI  features  are  based  on  certain  method  as¬ 
sumptions  and  design  principles  related  to  the  expected  use  of  the  method.  Those  assump¬ 
tions  and  principles  are  described  below. 

SCAMPI  is  a  Class  A  benchmarking  method. 

As  an  ARC  Class  A  method,  SCAMPI  can  be  used  to  generate  ratings  as  benchmarks  to  com¬ 
pare  maturity  levels  or  capability  levels  across  organizations.  SCAMPI  is  an  integrated  ap¬ 
praisal  method  that  can  be  applied  in  the  context  of  internal  process  improvement,  supplier 
selection,  or  process  monitoring.  As  a  benchmarking  method,  the  SCAMPI  emphasis  is  on  a 
rigorous  method  capable  of  achieving  high  accuracy  and  reliability  of  appraisal  results 
through  the  collection  of  objective  evidence  from  multiple  sources. 

Goal  ratings  are  a  function  of  the  extent  to  which  the  corresponding  practices  are  present  in 
the  planned  and  implemented  processes  of  the  organization. 

In  the  CMMI  models,  there  is  a  direct  relationship  between  goals  (specific  and  generic)  and 
the  practices  (specific  and  generic)  that  contribute  toward  achievement  of  those  goals.  Spe¬ 
cific  and  generic  goals  are  required  model  components;  specific  and  generic  practices  are 
expected  model  components,  in  that  alternative  practices  could  be  implemented  that  are 
equally  effective  in  achieving  the  intent  of  the  associated  goals. 


In  the  SCAMPI  method,  a  fundamental  premise  is  that  satisfaction  of  goals  can  be  deter¬ 
mined  only  upon  detailed  investigation  of  the  extent  to  which  each  corresponding  practice  is 
implemented,  for  each  sample  instance  used  as  a  basis  for  the  appraisal  (e.g.,  each  project). 

Additional  information  on  rating  goals  is  provided  in  “Data  Collection,  Rating,  and  Report¬ 
ing”  on  page  1-26. 
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The  aggregate  of  objective  evidence  provided  is  used  as  the  basis  for  determining  practice 
implementation. 

To  make  reasonable  judgments  regarding  an  organization’s  implemented  processes  relative  to 
the  CMMI  models,  appraisal  teams  base  their  judgments  on  the  collection  of  objective  evi¬ 
dence  for  each  specific  and  generic  practice  applicable  to  process  area  goals  within  the  ap¬ 
praisal  scope. 

Appraisal  teams  compare  the  objective  evidence  collected  against  the  corresponding  practices 
in  the  reference  model.  In  making  inferences  about  the  extent  to  which  practices  are  or  are 
not  implemented,  appraisal  teams  draw  upon  the  entire  model  document  to  understand  the 
intent  of  the  model,  and  use  it  as  the  basis  for  their  decisions.  This  includes  the  required  and 
expected  model  components  (i.e.,  generic  and  specific  goals,  generic  and  specific  practices), 
as  well  as  informative  material,  such  as  model  front  matter,  introductory  text,  glossary  defini¬ 
tions,  and  subpractices. 

Practice  implementation  at  the  organizational  unit  level  is  a  function  of  the  degree  of  prac¬ 
tice  implementation  at  the  instantiation  level  (e.g.,  projects). 

Practices  described  in  the  CMMI  models  are  abstractions  that  are  realized  by  their  implemen¬ 
tation  within  projects  and  organizations.  The  context  within  which  the  practice  is  applied 
drives  the  implementation.  The  details  of  the  implementation,  as  well  as  the  context  within 
which  the  practice  is  implemented,  are  referred  to  as  the  instantiation  of  the  process,  which 
may  occur  at  the  organizational  or  project  level. 

An  organizational  unit  is  the  part  of  an  organization  that  is  the  focus  of  an  appraisal.  An  or¬ 
ganizational  unit  operates  within  a  coherent  process  context  and  a  coherent  set  of  business 
objectives.  It  may  consist  of  a  set  of  related  projects.  (Refer  to  the  glossary  for  a  complete 
definition.) 

The  extent  to  which  an  organizational  unit  has  implemented  CMMI  model  practices  can  be 
determined  only  by  considering,  in  aggregate,  the  extent  to  which  those  practices  are  imple¬ 
mented  by  instantiations  of  the  process  (i.e.,  each  sample  project  considered  within  the  ap¬ 
praisal  scope).  This,  in  turn,  necessitates  the  consideration  of  objective  evidence  for  each  in¬ 
stantiation,  for  each  model  practice  within  the  appraisal  scope. 

Appraisal  teams  are  obligated  to  seek  and  consider  objective  evidence  of  multiple  types  in 
determining  practice  implementation  and  goal  satisfaction. 

The  SCAMPI  method  is  data  oriented,  in  that  decisions  on  practice  implementation  and  rat¬ 
ing  are  made  based  upon  the  aggregate  of  objective  evidence  available  to  the  appraisal  team. 
Multiple  types  of  objective  evidence  must  be  considered;  these  are  described  in  “Objective 
Evidence  Sources”  on  page  1-23.  Artifacts  indicative  of  practice  implementation  are  a  re- 
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quirement  of  the  SCAMPI  method.  Face-to-face  interviews  are  required  to  ensure  that  the 
documentation  is  reflective  of  the  actual  organizational  process  implementation,  and  to  pre¬ 
clude  rating  judgments  being  made  solely  on  the  basis  of  artifacts.  The  SCAMPI  method  es¬ 
tablishes  minimum  requirements,  described  in  “Data  Collection,  Rating,  and  Reporting”  on 
page  1-26,  for  the  extent  to  which  objective  evidence  from  face-to-face  interviews  must  be 
collected  for  model  practices  to  corroborate  other  sources  of  objective  evidence  prior  to  rat¬ 
ing  goals. 

Verification  vs.  Discovery 

If  an  organization  has  in  place  assets,  mechanisms,  and  objective  evidence  that  substantiate 
its  implementation  of  model  practices,  it  is  in  the  organization’s  best  interest  to  share  that 
knowledge  to  ensure  that  the  appraisal  team  obtains  a  complete  and  accurate  understanding  of 
the  organization’s  implementation  of  model  practices.  Many  organizations  capture  this  under¬ 
standing  through  assets  such  as  traceability  and  mapping  tables  from  the  model  to  their  or¬ 
ganizational  processes  and  project  instantiations.  Implementation  of  the  model  within  the  or¬ 
ganization  may  be  further  reinforced  through  additional  mechanisms,  such  as; 

•  documentation  (e.g.,  policies,  process  descriptions,  project  plans) 

•  verification  and  oversight  activities  (e.g.,  internal  appraisals,  audits,  reviews,  status  re¬ 
ports) 

•  tools  and  resources  (e.g.,  databases,  measurement  repositories,  configuration  manage¬ 
ment  tools) 

If  assets  such  as  these,  or  indicators  of  the  existence  of  the  assets,  are  made  available  to  the 
appraisal  team,  this  leaves  the  appraisal  team  the  task  of  verifying  whether  the  objective  evi¬ 
dence  provided  is  adequate  for  substantiation  of  practice  implementation.  This  verification- 
based  approach  is  in  contrast  to  the  more  difficult,  error  prone,  and  time-consuming  task  of 
investigating  each  practice  to  discover  the  objective  evidence  needed  to  substantiate  imple¬ 
mentation.  In  a  verification-based  approach,  both  the  organizational  unit  and  the  appraisal 
team  have  a  clearer  picture  of  what  artifacts  are  available  and  what  might  still  be  needed, 
thereby  minimizing  the  amount  of  further  investigation  necessary  in  the  form  of  interviews 
and  additional  documentation  requests.  The  verification-based  approach  thus  facilitates  ap¬ 
praisals  that  are  accurate,  repeatable,  efficient,  and  that  provide  meaningful  results;  in  other 
words,  appraisals  that  satisfy  the  essential  method  characteristics  described  in  “Method  Ob¬ 
jectives  and  Characteristics”  on  page  1-15. 


Whereas  some  legacy  appraisal  methods  encouraged  organizations  to  provide  such  assets,  the 
en^hasis  is  strengthened  further  in  the  SCAMPI  method,  which  is  designed  on  the  assump¬ 
tion  that  relevant  objective  evidence  is  available  for  review  in  advance  of  the  on-site  period. 
This  assumption  is  typically  discussed  with  the  appraisal  sponsor  and  his/her  representatives 
during  development  of  the  appraisal  plan.  A  key  milestone  in  the  appraisal  process  is  a  review 
prior  to  the  appraisal  on-site  period  to  determine  readiness  to  proceed  with  a  verification- 
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based  appraisal  as  planned.  If  the  appraised  organization  has  not  provided  objective  evidence 
of  sufficient  quality  and  completeness  to  enable  a  verification-based  appraisal,  the  appraisal 
plan  may  need  to  be  renegotiated  to  reflect  the  additional  effort  that  must  be  undertaken  for 
the  appraisal  team  to  search  for  and  discover  that  objective  evidence  during  the  on-site  pe¬ 
riod. 

Objective  Evidence  Sources 

The  SCAMPI  method  provides  for  the  collection  of  data  from  the  following  sources: 

•  Instruments  -  Written  information  relative  to  the  organizational  unit’s  implementation  of 
CMMI  model  practices.  This  can  include  assets  such  as  questionnaires,  surveys,  or  an  or¬ 
ganizational  mapping  of  CMMI  model  practices  to  its  corresponding  processes.  See  “In¬ 
struments  and  Tools”  on  page  1-29  for  additional  information  on  the  use  of  appraisal  in¬ 
struments. 

•  Presentations  -  Information  prepared  by  the  organization  and  delivered  visually  or  ver¬ 
bally  to  the  appraisal  team  to  describe  organizational  processes  and  implementation  of 
CMMI  model  practices.  This  typically  includes  such  mechanisms  as  orientation  or  over¬ 
view  briefings,  and  demonstrations  of  tools  or  capabilities. 

•  Documents  -  Artifacts  reflecting  the  implementation  of  one  or  more  model  practices. 
These  typically  include  organizational  policies,  procedures,  and  implementation-level  ar¬ 
tifacts.  Documents  may  be  available  in  hardcopy  or  softcopy,  or  may  be  accessible  via 
hyperlinks  in  a  web-based  environment. 

•  Interviews  -  Face-to-face  interaction  with  those  implementing  or  using  the  processes 
within  the  organizational  unit.  Interviews  are  typically  held  with  various  groups  or  indi¬ 
viduals,  such  as  project  leaders,  managers,  and  practitioners.  A  combination  of  formal 
and  informal  interviews  may  be  held,  using  interview  scripts  or  exploratory  questions  de¬ 
veloped  to  elicit  the  information  needed. 

Using  multiple  data-gathering  mechanisms  improves  the  depth  of  understanding  and  enables 
corroboration  of  the  data. 

Focused  Investigation 

Due  to  the  quantity  of  CMMI  model  practices  that  must  be  investigated  and  the  SCAMPI 
rules  for  collection  of  objective  evidence  to  ensure  sufficient  coverage  of  these  practices  for 
rating  (see  “Data  Collection,  Rating,  and  Reporting”  on  page  1-26),  it  is  crucial  that  appraisal 
teams  apply  efficient  techniques  for  the  collection  and  management  of  appraisal  data.  This 
focus  on  efficient  data  management  practices  is  integral  to  SCAMPI  method  concepts,  and  is 
emphasized  throughout  the  appraisal  process.  The  term  “focused  investigation”  is  used  in 
SCAMPI  to  describe  this  concept  of  optimized  investment  of  appraisal  resources.  Essentially, 
this  can  be  described  at  a  top  level  using  the  following  data  collection  and  investigation  para¬ 
digms: 
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•  Understand  what  objective  evidence  is  available,  and  how  it  contributes  toward  imple¬ 
mentation  of  model  practices  within  the  appraisal  scope. 

•  Continually  consolidate  data  to  determine  progress  toward  sufficient  coverage  of  model 
practices. 

•  Focus  appraisal  resources  by  targeting  those  areas  for  which  further  investigation  is 
needed  to  collect  additional  data  or  verify  the  set  of  objective  evidence. 

•  Avoid  unnecessary  or  duplicated  effort  that  does  not  contribute  additional  information 
toward  achievement  of  sufficient  coverage  or  toward  obtaining  significantly  greater  con¬ 
fidence  in  the  appraisal  results.  For  example,  keep  interviews  efficient  by  asking  further 
questions  only  about  practices  for  which  sufficient  data  has  not  already  been  obtained. 

These  concepts,  derived  from  the  best  practices  of  experienced  lead  appraisers,  are  primary 
mechanisms  used  to  achieve  efficient  appraisal  performance  by  emphasizing  the  placement  of 
appraisal  team  effort  where  it  is  most  needed.  This  begins  with  the  initial  collection  and 
analysis  of  objective  evidence  from  the  organizational  unit.  This  analysis  can  be  used  to  de¬ 
termine  the  adequacy  and  completeness  of  the  provided  objective  evidence,  and  to  identify 
the  extent  to  which  further  investigation  is  necessary.  The  appraisal  team’s  inventory  of  ob¬ 
jective  evidence  can  be  annotated  to  identify  practices  that  are  strongly  supported,  or  those 
that  need  further  clarification.  This  knowledge  can  be  used  as  the  basis  for  determining  find¬ 
ings  that  affect  appraisal  outcomes. 


As  the  appraisal  process  progresses,  the  appraisal  team  aggregates  and  synthesizes  additional 
objective  evidence  from  process  instantiations,  and  uses  this  to  draw  inferences  about  the 
overall  implementation  within  the  organizational  unit.  Wherever  there  are  shortcomings  in 
the  appraisal  team’s  understanding  of  the  organizational  unit’s  implementation  of  model  prac¬ 
tices,  data  collection  strategies  can  be  determined  to  probe  for  and  obtain  additional  informa¬ 
tion.  For  example,  cases  where  the  objective  evidence  is  missing,  unclear,  or  insufficient 
might  be  addressed  through  additional  documentation  requests  or  by  generating  focused 
questions  for  specific  interview  participants.  By  maintaining  a  current  inventory  of  the  status 
of  the  appraisal  objective  evidence  and  prioritizing  areas  where  additional  information  is  still 
needed,  these  focused  investigation  approaches  can  be  continuously  and  iteratively  applied  to 
narrow  remaining  gaps  and  converge  on  sufficient  coverage  for  proceeding  with  rating. 

Additional  information  on  focused  investigation  and  continuous  consolidation  concepts  can 
be  found  in  Appendix  C. 

Practice  Implementation  Indicators 

The  fundamental  idea  of  Practice  Implementation  Indicators  (PIIs)  is  that  the  conduct  of  an 
activity  or  the  implementation  of  a  practice  results  in  “footprints” — evidence  that  provides  a 
basis  for  verification  of  the  activity  or  practice. 
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In  SCAMPI,  Practice  Implementation  Indicators  are  the  necessary  consequence  of  imple¬ 
menting  CMMI  model  practices.  For  example,  the  establishment  of  an  artifact,  such  as  a 
document,  is  often  an  expected  outcome  resulting  from  implementation  of  a  model  practice. 
Other  indicators  may  indirectly  substantiate  implementation  of  the  practice,  such  as  evidence 
of  a  status  meeting  or  peer  review  being  held.  Members  of  the  organizational  unit  may  affirm 
through  questionnaires  or  interviews  that  the  practice  is  implemented.  These  are  all  potential 
“footprints”  that  can  be  used  as  objective  evidence  to  verify  and  substantiate  implementation 
of  model  practices. 

SCAMPI  characterizes  PIIs  according  to  the  indicator  types  described  in  Table  1-9, 


Table  T9:  Practice  Implementation  Indicator  Types 


Indicator  Type 

Description 

Examples 

Direct  artifacts 

The  tangible  outputs  resulting  directly 
from  implementation  of  a  specific  or  ge¬ 
neric  practice.  An  integral  part  of  verify¬ 
ing  practice  implementation.  May  be  ex¬ 
plicitly  stated  or  implied  by  the  practice 
statement  or  associated  informative  mate¬ 
rial. 

Typical  work  products  listed 
in  CMMI  model  practices 

Target  products  of  an  “Estab¬ 
lish  and  Maintain”  specific 
practice 

Documents,  deliverable 
products,  training  materials, 
etc. 

Indirect  artifacts 

Artifacts  that  are  a  consequence  of 
performing  a  specific  or  generic  practice 
or  that  substantiate  its  implementation, 
but  which  are  not  the  purpose  for  which 
the  practice  is  performed.  This  indicator 
type  is  especially  useful  when  there  may 
be  doubts  about  whether  the  intent  of  the 
practice  has  been  met  (e.g.,  an  artifact 
exists  but  there  is  no  indication  of  where 
it  came  from,  who  worked  to  develop  it, 
or  how  it  is  used). 

Typical  work  products  listed 
in  CMMI  model  practices 

Meeting  minutes,  review  re¬ 
sults,  status  reports 

Performance  measures 

Affirmations 

i 

Oral  or  written  statements  confirming  or 
supporting  implementation  of  a  specific 
or  generic  practice.  These  are  usually 
provided  by  the  implementers  of  the 
practice  and/or  internal  or  external  cus¬ 
tomers,  but  may  also  include  other  stake¬ 
holders  (e.g.,  managers,  suppliers). 

Questionnaire  responses 

Interviews 

Presentations 

Appraisal  teams  are  obligated  to  seek  objective  evidence  of  each  of  these  types  as  a  prerequi¬ 
site  to  formulating  characterizations  of  practice  implementation.  The  indicator  types  that  will 
be  most  appropriate  to  reflect  practice  implementation  will  vary  according  to  the  context  in 
which  the  process  is  implemented,  as  well  as  the  practice  itself.  The  appraisal  team  should 
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consider  all  aspects  of  the  process  context,  such  as  project  size  and  duration,  organizational 
culture,  application  domain,  customer  market,  and  so  on,  in  determining  the  appropriateness 
and  sufficiency  of  indicators.  For  example,  the  level  of  detail  necessary  for  a  work  break¬ 
down  structure  will  differ  widely  for  a  1 -person,  2-week  maintenance  effort  as  opposed  to  a 
100-person,  multi-year,  mission-critical,  new  product  development. 

An  example  of  how  PIIs  for  each  of  these  types  might  be  used  in  verifying  implementation  of 
a  model  practice  is  depicted  in  Figure  I-l. 


PP  SP1.1 


Indirect  work  product: 


Indirect  work  product:  Affirmation: 

Project  estimates  aligned  How  is  the  WBS  used? 

with  WBS  elements  How  are  estimates  generated? 


Figure  F1:  Exam  pie  of  PH  Use 

Appraisal  teams  collect  and  organize  data  according  to  these  indicator  types.  The  SCAMPI 
method  defines  rules  and  guidelines  (described  in  “Data  Collection,  Rating,  and  Reporting” 
below)  about  the  amount  of  data  that  must  be  collected  for  each  of  these  indicator  types.  A 
combination  of  objective  evidence  according  to  these  indicator  types  is  necessary  to  corrobo¬ 
rate  multiple  sources  of  data  that  may  be  available  for  each  practice,  and  to  obtain  confidence 
in  the  accuracy  of  the  data  collected.  For  reasons  that  are  evident,  an  over-reliance  on  one 
type  of  objective  evidence  or  another  is  undesirable.  Too  much  dependence  on  artifacts  could 
result  in  the  perception  that  the  appraisal  was  a  “paper  review”  and  not  truly  indicative  of 
organizational  and/or  project  behavior.  An  over-reliance  on  affirmations  could  be  criticized  as 
not  truly  objective  or  repeatable.  Therefore,  the  SCAMPI  method  requires  a  balance  across 
these  types  of  objective  evidence. 


Appendix  B  contains  additional  detailed  discussion  of  PIIs  and  indicator-based  appraisals. 

Data  Collection,  Rating,  and  Reporting 

The  appraisal  team  follows  a  consensus-based,  structured  process  to  synthesize  and  transform 
information  collected  from  the  sources  described  in  “Objective  Evidence  Sources”  on  page  I- 
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23.  Data  from  these  sources  are  collected  and  considered  in  several  discrete  data-gathering 
sessions,  either  as  integrated  appraisal  team  activities  or  by  subsets  of  the  team  organized  into 
mini-teams  operating  in  parallel.  Mini-teams  are  typically  organized  around  related  process 
areas,  with  mini-team  members  assigned  by  the  appraisal  team  leader  on  the  basis  of  their 
individual  experience,  knowledge,  and  skills. 


The  SCAMPI  data  transformation  and  rating  process  is  depicted  in  Figure  1-2. 


Level  of 
Consensus 


Capability  Level  and/or 
Maturity  Level  Ratings 


Full  Team 


Goal  Satisfaction  Ratings 


Practice  Implementation  Characterizations 
(organizational  unit  level) 


Practice  Implementation  Characterizations 
(practice  instantiation  level) 


Full  Team 

Full  Team 

Mini-Team 


Figure  1-2:  SCAMPI  Rating  Process 

Team  members  review  objective  evidence  provided  by  the  organizational  unit  and  identify 
PIIs  relative  to  the  reference  model  practices.  These  PIIs  are  categorized  as  direct  artifacts, 
indirect  artifacts,  or  affirmations,  as  described  in  “Practice  Implementation  Indicators”  on 
page  1-24,  and  are  added  to  the  team’s  PII  inventory. 

Areas  of  significant  strength  or  weakness  observed  relative  to  the  implementation  of  model 
specific  or  generic  practices  are  recorded  in  written  observations.  Observations  are  generated 
primarily  for  weaknesses,  or  “gaps,”  of  the  implementation  compared  to  the  intent  of  a  model 
practice.  Observations  of  strengths  should  be  reserved  for  implemented  practices  that  are  par¬ 
ticularly  effective  and  are  candidates  for  inclusion  in  aggregated  findings.  “Gratuitous” 
strengths  that  simply  reflect  a  sufficient  implementation  of  a  practice  can  produce  substantial 
data  management  overhead  that  does  not  contribute  toward  generation  of  findings;  these  are 
more  effectively  captured  as  indicators  in  the  appraisal  team’s  PII  inventory.  Observations 
may  also  be  generated  for  alternative  practices,  which  are  acceptable  alternatives  to  imple¬ 
menting  one  or  more  model  practices  that  contribute  equivalently  to  the  satisfaction  of  proc¬ 
ess  area  goals. 
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Characterizing  Practices 

Verification  of  Practice  Implementation  Indicators  continues  in  this  way  at  the  instantiation 
level  until  sufficient  objective  evidence  has  been  obtained  to  characterize  the  implementation 
of  a  specific  or  generic  practice.  Sufficiency  at  the  practice  level  for  an  instantiation  is  deter¬ 
mined  when  direct  artifacts  covering  the  intent  of  the  practice  have  been  verified  for  the  prac¬ 
tice  and  corroborated  by  indirect  artifacts  or  affirmations.  Consensus  is  obtained  at  the  mini¬ 
team  level  on  the  sufficiency  of  instantiation-level  practice  implementation  indicators  and 
accuracy  of  observations  of  strengths  and  weaknesses. 


Based  upon  the  practice  implementation  data  for  a  process  instantiation,  the  appraisal  team 
(or  typically  a  mini-team)  assigns  values  to  characterize  the  extent  to  which  the  CMMI  model 
practice  is  implemented.  Each  practice  is  characterized  as  Fully  Implemented  (FI),  Largely 
Implemented  (LI),  Partially  Implemented  (PI),  or  Not  Implemented  (NI).  The  intent  of  this 
characterization  is  to  effectively  summarize  the  appraisal  team’s  judgment  of  practice  imple¬ 
mentation  as  a  mechanism  to  identify  where  team  judgment  is  most  needed,  and  to  prioritize 
areas  where  further  investigation  or  corroboration  may  be  necessary.  These  characterization 
values  are  an  aid,  not  a  replacement,  for  the  observations  recorded  for  strengths  and  weak¬ 
nesses,  which  are  used  as  a  basis  for  rating  decisions. 

Upon  assigning  characterization  values  for  a  given  model  practice  for  each  instantiation,  the 
characterization  values  are  aggregated,  using  full  appraisal  team  consensus,  to  the  organiza¬ 
tional  unit  level.  Observations  reflecting  strengths  and  weaknesses  across  instantiations  are 
similarly  aggregated  to  the  organizational  unit  level,  and  form  the  basis  for  rating.  Where 
team  judgment  is  necessary  to  characterize  practice  implementation,  these  decisions  are  made 
considering  factors  such  as  the  mix  of  practice  characterizations,  the  reason  for  the  instantia¬ 
tion-level  characterizations,  and  the  severity  of  the  associated  weaknesses  (in  aggregate). 

Tracking  Progress 

The  appraisal  team  uses  focused  investigation  techniques  (see  “Focused  Investigation”  on 
page  1-23)  to  track  progress  toward  sufficient  coverage  necessary  for  rating  process  area 
goals  within  the  appraisal  scope.  Revisions  to  the  data  collection  plan  may  be  necessary  to 
ensure  that  adequate  objective  evidence  is  obtained  from  each  instantiation  (e.g.,  project)  for 
each  specific  and  generic  practice  within  the  reference  model  scope  of  the  appraisal.  If  insuf¬ 
ficient  objective  evidence  is  available,  the  data  collection  plan  may  be  revised  to  conduct  ad¬ 
ditional  data-gathering  sessions.  Focused  investigation  techniques  can  be  used  to  ensure  pro¬ 
gress  toward  sufficient  coverage  of  model  practices,  goals,  and  process  areas  within  the 
appraisal  scope. 
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Generating  Findings 

Strengths  and  weaknesses  identified  across  instantiations  are  synthesized  and  aggregated  to 
statements  of  preliminary  findings,  expressed  at  the  organizational  unit  level.  These  are  often 
organized  at  the  level  of  process  area  goals  using  common  themes.  Preliminary  findings  are 
provided  to  the  organizational  unit  for  validation;  the  mechanisms  and  timeframe  used  for 
this  may  vary  across  the  appraisal  modes  of  usage  .(internal  process  improvement,  supplier 
selection,  process  monitoring).  During  this  activity,  the  appraisal  team  is  still  in  the  process  of 
collecting  data  to  ensure  that  an  accurate  understanding  of  the  organizational  process  imple¬ 
mentation  is  obtained.  Feedback  from  the  participants  in  the  appraisal  is  used  to  validate  the 
preliminary  findings,  and  may  result  in  additional  observations  or  revision  to  the  findings. 

The  appraisal  team  may  also  request  additional  data  sources  for  areas  where  their  understand¬ 
ing  of  the  organization’s  implementation  of  model  practices  is  insufficient.  Final  findings  are 
generated  based  on  the  complete,  validated  set  of  appraisal  data  (i.e.,  findings,  aggregated 
strengths  and  weaknesses,  and  inventory  of  PIIs). 

Generating  Ratings 

Ratings  are  generated  based  on  the  set  of  validated  appraisal  data.  At  a  minimum,  ratings  are 
generated  for  each  of  the  process  area  generic  and  specific  goals  within  the  appraisal  refer¬ 
ence  model  scope.  Ratings  may  also  be  generated  for  process  areas,  capability  levels,  or  ma¬ 
turity  levels  if  desired  by  the  appraisal  sponsor.  Maturity  level  ratings  and/or  capability  level 
ratings  are  based  on  the  definitions  of  capability  levels  and  maturity  levels  in  the  CMMI 
models.  Refer  to  Process  Description  2.4,  “Generate  Appraisal  Results,”  for  additional  infor¬ 
mation  about  SCAMPI  rating  processes. 

Reporting  Results 

The  results  of  the  appraisal  are  reported  to  the  appraisal  sponsor.  For  source  selection  and 
process  monitoring  contexts,  these  results  are  also  provided  to  the  appraised  organization;  the 
mechanisms  and  timeframe  used  for  this  may  be  subject  to  acquisition  or  contractual  restric¬ 
tions.  An  appraisal  record  is  generated  and  provided  to  the  sponsor,  documenting  further  in¬ 
formation  regarding  the  appraisal. 

A  subset  of  this  data  is  provided  to  the  CMMI  Steward  for  the  purposes  of  quality  control  and 
the  collection  of  appraisal  measures  for  reporting  to  the  appraisal  community.  The  appraisal 
data  to  be  provided  is  defined  by  the  Steward  separately  from  this  document  to  allow  for  con¬ 
tinuous  improvement  of  appraisal  reporting  apart  from  the  CMMI  Product  Suite. 

Instruments  and  Tools 

Instruments  are  artifacts  used  in  an  appraisal  for  the  collection  and  presentation  of  data.  In¬ 
struments  are  provided  by  the  organizational  unit  to  inform  the  appraisal  team  about  the 
processes  implemented  in  the  organization  and  how  they  relate  to  the  CMMI  reference  mod- 
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els.  Instruments  can  take  various  forms,  including  questionnaires,  surveys,  site  orientation 
packets,  and  mappings  from  CMMI  practices  to  the  organizational  or  project  processes. 


The  SCAMPI  method  does  not  require  any  particular  instrument  or  presentation  format,  only 
that  an  instrument  be  used.  Instruments  can  be  used  most  elfectively  if  they  provide  the  ap¬ 
praisal  team  with  an  in-depth  understanding  of  the  organizational  implementation  of  the 
model,  on  a  practice-level  basis  for  each  instantiation  to  be  investigated  in  the  appraisal.  In¬ 
struments  also  often  provide  an  opportunity  for  the  organizational  unit  to  provide  a  self¬ 
characterization  of  their  implemented  processes,  identify  applicable  substantiating  objective 
evidence,  and  specify  any  additional  comments  that  might  be  useful  in  understanding  the  im¬ 
plemented  processes.  Used  in  this  manner,  instruments  can  support  the  SCAMPI  method  em¬ 
phasis  on  verification-based  appraisals  and  minimize  the  need  for  on-site  discovery  of  objec¬ 
tive  evidence  (see  “Verification  vs.  Discovery”  on  page  1-22),  thus  helping  to  facilitate 
efficient  appraisal  performance. 


As  described  in  “Practice  Implementation  Indicators”  on  page  1-24,  the  SCAMPI  method 
emphasizes  the  use  of  PIIs.  Organizations  may  provide  as  input  to  the  appraisal  a  PII  data¬ 
base  (PIIDB),  with  a  mapping  of  model  practices  to  corresponding  processes  and  objective 
evidence  that  can  be  used  to  verify  practice  implementation.  It  is  anticipated  that  many  or¬ 
ganizations  will  have  existing  assets  in  place  that  reflect  their  process  implementation  and 
mapping  to  CMMI  model  practices.  These  instruments  can  be  used  as  a  source  of  appraisal 
data  in  much  the  same  way  as  a  PIIDB.  The  collection  of  these  model  mappings  and  indica¬ 
tors  can  be  a  valuable  resource  for  process  improvement  at  the  organization  and  project  lev¬ 
els,  and  a  rich  source  of  data  for  process  appraisals  using  a  variety  of  Class  A,  B,  and  C  ap¬ 
praisal  methods. 


It  is  recommended  that  a  member  of  the  appraisal  team  facilitate  the  entry  of  data  into  in¬ 
struments  where  feasible,  to  ensure  that  appropriate  data  are  obtained.  This  can  help  the  ap¬ 
praised  organization  clarify  or  interpret  the  intent  of  the  model  practices,  understand  what 
data  are  expected,  and  focus  the  responses.  The  entry  of  either  too  much  or  too  little  data  into 
instruments  can  be  problematic  for  both  the  appraisal  team  and  the  appraised  organization 
and  result  in  inefficient  use  of  resources. 


Effective  management  of  appraisal  data  is  a  significant  challenge  that  can  be  simplified  with 
the  use  of  automated  tools.  The  CMMI  Steward  provides  a  rudimentary  toolkit  to  Lead  Ap¬ 
praisers  that  can  be  used  to  collect  practice-leyel  questionnaire  data;  characterize,  consoli¬ 
date,  and  summarize  responses;  and  record  observations  based  on  these  responses  where  ap¬ 
propriate.  Several  vendor  tools  are  also  available  in  the  marketplace.  The  choice  of  tools  is 
largely  one  of  personal  preference;  some  experienced  appraisers  prefer  manual  techniques, 
such  as  wall  charts,  to  record  observations  and  findings. 
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Effective  Team  Practices 

Appraisal  team  dynamics  and  effective  group  techniques  contribute  significantly  to  the  ability 
to  conduct  SCAMPI  appraisals.  The  appraisal  team  leader  can  help  focus  team  activities  so 
that  effort  is  spent  wisely  toward  achievement  of  method  requirements  and  appraisal  objec¬ 
tives.  “Method  Performance”  on  page  1-18  contains  several  efficiency  ideas  identified  by  the 
appraisal  community  as  potential  areas  for  improvement.  SCAMPI  features  encourage  effec¬ 
tive  team  practices  that,  with  the  support  of  the  appraisal  team  leader,  can  address  some  of 
these  issues.  This  includes  areas  such  as: 

•  Verification-based  approach  -  Verification  of  PIIs  provided  as  objective  evidence  by  the 
organization  in  advance  of  the  appraisal  is  emphasized  to  reduce  the  extent  of  data  that 
must  be  obtained  through  discovery  techniques  during  the  on-site  period.  Even  in  this 
case,  it  is  recommended  that  the  entry  of  PII  data  by  the  organizational  unit  be  facilitated 
to  ensure  that  an  appropriate  and  useful  set  of  objective  evidence  is  available;  too  much 
data  that  is  not  useful  is  just  as  great  a  problem  as  too  little  data.  (See  “Verification  vs. 
Discovery”  on  page  1-22.) 

•  Reduced  crafting  of  observations  -  In  an  indicator-based  appraisal,  greater  emphasis  is 
placed  on  verification  of  PIIs,  and  there  is  less  need  overall  for  crafting  notes  and  obser¬ 
vations.  Observations  need  not  be  generated  simply  to  acknowledge  satisfactory  imple¬ 
mentations  or  existence  of  artifacts,  but  can  focus  more  on  identifying  weaknesses  or  sig¬ 
nificant  strengths  that  can  be  expected  to  be  included  in  the  findings. 

•  Consensus  -  Mini-teams  are  given  the  authority  to  reach  consensus  on  practice  imple¬ 
mentation  at  the  instantiation  level;  full  team  consensus  is  required  for  aggregation  to  the 
organizational  unit  level.  (See  Figure  2.2.6-1.)  The  characterization  of  practice  imple¬ 
mentation  (FI,  LI,  PI,  NI;  see  “Data  Collection,  Rating,  and  Reporting”  on  page  1-26)  can 
also  help  facilitate  consensus  on  whether  implementations  satisfy  model  intent,  either  at 
the  instantiation  or  organizational  unit  level.  The  consensus,  discussion,  and  decision¬ 
making  processes  used  by  the  appraisal  team  can  be  significant  sources  of  inefficiency  if 
not  monitored  closely. 

•  Corroboration  -  Corroboration  is  built  into  the  method  through  requirements  for  multiple 
types  of  objective  evidence  (direct  artifacts,  indirect  artifacts,  affirmations).  (See  Section 
2.2.5.) 

•  Effective  data  management  -  The  SCAMPI  method  provides  ways  to  collect,  organize, 
and  manage  appraisal  data  efficiently,  and  to  facilitate  the  team  decisions  that  must  be 
made  based  on  the  set  of  objective  evidence.  The  focused  investigation  techniques  de¬ 
scribed  in  Section  2.2.4  can  help  keep  the  team  oriented  on  what  objective  evidence  has 
been  collected,  what  remains  to  be  collected,  and  how  it  will  be  collected.  This  can  be 
greatly  enhanced  through  the  use  of  automated  support  tools.  A  thorough  understanding 
of  progress  toward  sufficiency  of  coverage  can  help  focus  data  collection.  Interviews,  in 
particular,  can  be  shortened  by  focusing  on  specific  data  collection  needs. 

Several  additional  effective  team  practices  are  targeted  toward  specific  subsets  of  the  ap¬ 
praisal,  and  are  included  as  suggested  implementation  or  tailoring  options  within  individual 
process  descriptions  in  Part  II. 
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Method  Description 

This  section  provides  an  overview  of  the  SCAMPI  method  architecture,  including  appraisal 
phases,  processes,  and  activities.  These  descriptions  are  high-level  abstractions  of  the  process 
descriptions  contained  in  Part  II  of  this  document. 

A  summary  of  the  SCAMPI  method  processes  and  activities  for  each  of  the  three  appraisal 
phases  is  contained  in  Tables  I-IO  through  1-12. 


The  interactions  between  the  processes  are  depicted  in  the  process  flow  diagrams  in  Figures 
1-3  through  1-5.  These  diagrams  show  the  work  products  that  are  inputs  and  outputs  at  the 
process  level  for  accomplishing  the  purpose  of  the  appraisal.  Additional  analysis  was  done  to 
ensure  that  the  activities  within  each  process  use  and  provide  the  inputs  and  outputs  of  the 
process.  However,  that  detailed  analysis  is  not  presented  here. 

The  process  flows  generally  show  summarized  and  completed  products.  For  instance,  the  ap¬ 
praisal  input  generated  by  Analyze  Requirements  initially  is  provided  to  the  Develop  Ap¬ 
praisal  Plan  process  with  some  elements  missing  that  are  generated  in  other  processes.  These 
flow  back  to  Analyze  Requirements  in  the  appraisal  plan.  The  final  appraisal  input  as  coordi¬ 
nated  with  the  sponsor  is  then  produced  as  a  completed  product.  Additional  administrative 
and  support  products,  such  as  appraisal  checklists,  will  be  produced  but  are  not  included  in 
these  diagrams. 


The  time  sequences  of  appraisals  are  also  shown  in  the  process  flow  diagrams.  Figures  1-6 
and  1-7  show  nominal  schedules  for  conduct  of  appraisals  in  both  assessment  and  evaluation 
modes.  There  are  several  differences  between  and  tailoring  options  within  each  of  these 
schedules.  They  are  examples  and  are  not  intended  to  be  requirements. 


For  assessments,  the  preparation  of  participants  (“Prep”  in  the  diagram)  can  be  at  any  time 
between  the  identification  of  the  participants  and  the  administration  of  the  instruments.  If  a 
set  of  PIIs  assembled  by  the  organization  for  previous  appraisals  is  chosen  as  the  instrument, 
this  would  be  at  the  beginning  of  data  collection.  If  an  additional  completion  of  instruments 
beyond  those  provided  in  the  initial  objective  evidence  is  not  required,  this  preparation  could 
be  delayed  to  just  prior  to  interviews.  Other  options  include  but  are  not  limited  to  the  timing 
of  team  selection,  number  and  timing  of  readiness  reviews,  and  sequence  of  artifact  and  in¬ 
terview  activities. 


For  evaluations,  there  are  several  differences  in  sequence  from  assessments.  For  instance,  the 
analysis  of  initial  objective  evidence  occurs  after  the  organizations  have  responded  to  the  re¬ 
quest  for  data,  which  in  turn  follows  the  completion  of  the  Data  Collection  Plan.  Another  dif¬ 
ference  is  the  delay  of  the  delivery  of  the  appraisal  results  until  after  all  organizations  have 
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been  appraised.  An  example  of  timing  options  is  that  the  identification  of  the  appraisal  team 
leader  may  be  delayed  until  near  the  end  of  the  planning  activities.  This  ordering  of  events 
must  be  accommodated  by  the  plan;  for  example,  completion  of  the  Analyze  Requirements 
and  Develop  Appraisal  Plan  processes  must  be  rescheduled  to  allow  the  appraisal  team  leader 
to  approve  the  appraisal  input  and  the  appraisal  plan. 
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Objective  Evidence  sources  of  data,  tools  and  technologies  to  be  used,  and  contingencies  j  ^  2  Prepare  Data  Collection  Plan 

to  manage  risk  of  insufficient  data.  *  ^ 

1.5.3  Replan  Data  Collection  (if  needed) 


Table  1-11:  SCAMPI  Phase  Summary:  Conduct  Appraisal 
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Table  1-12:  SCAMPI  Phase  Summary:  Report  Results 
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Prepared  Team 


Figure  1-4:  Process  Flows,  Conduct  Appraisal  Processes 


Figure  1-5:  Process  Flows,  Report  Results  Processes 


Nominal  Assessment  Schedule  View 


Figure  1-6:  Nominal  Schedule  for  Assessment  Mode 


Nominal  Evaluation  Schedule  View 
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1.1  Analyze  Requirements 


Purpose 

Entry  Criteria 

Inputs 

Activities 

Outputs 

Outcome 

Exit  Criteria 


Understand  the  business  needs  of  the  organization  for  which  the  appraisal  is 
being  requested.  The  appraisal  team  leader  will  collect  information  and  help 
the  appraisal  sponsor  match  appraisal  objectives  with  their  business 
objectives. 


•  An  appraisal  sponsor  has  decided  that  a  SCAMPI  appraisal  should  be 
performed. 

•  People  who  can  provide  statements  of  requirements  for  the  appraisal  are 
available. 


•  Sponsor 

•  Initial  requirements  and  constraints 

•  Process-related  legacy  information 


1.1.1  Determine  Appraisal  Obj  ectives 

1.1.2  Determine  Appraisal  Constraints 

1.1.3  Determine  Appraisal  Scope 

1.1.4  Determine  Outputs 

1.1.5  Obtain  Commitment  to  Appraisal  Input 


The  appraisal  input 


The  decision  to  proceed  with  the  appraisal  based  on  a  shared  understanding  of 
the  appraisal  objectives,  constraints,  outputs,  and  scope. 


•  Initial  contact  between  the  appraisal  sponsor  and  authorized  SCAMPI 
Lead  Appraiser  has  occurred. 

•  The  Lead  Appraiser  has  been  given  access  to  members  of  the  sponsoring 
organization. 

•  The  appraisal  input  has  been  approved  by  the  appraisal  sponsor  and 
placed  under  change  management. 


Continued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Key  Points 


Toois  and 
Techniques 


Metrics 


Verification  and 
Validation 

Records 

Tailoring 


At  this  early  stage  in  the  process,  gathering  information  that  supports  good 
planning  is  most  important.  Often,  the  appraisal  team  leader  must  educate 
members  of  the  sponsor’s  organization  in  the  purpose  and  role  of  appraisals. 


Collaborative  consultation  between  the  appraisal  team  leader  and  the  appraisal 
sponsor  is  important  in  this  activity.  The  appraisal  team  leader  may  be  able  to 
simply  interview  the  sponsor  to  get  the  needed  information  and  reach 
agreements.  In  some  settings,  a  series  of  meetings  with  different  stakeholders 
may  be  needed  to  elicit  and  build  consensus  on  the  business  needs  that  can  be 
met  through  a  SCAMPI  appraisal. 

Understanding  the  history  of  appraisals  in  the  organization,  especially  the 
organizational  and  model  scope  of  past  appraisals,  is  important  for 
understanding  the  requirements  for  the  appraisal  under  consideration.  The 
choices  sponsors  make  about  appraisal  scope  are  often  tied  to  their 
(sometimes-unstated)  priorities  for  process  improvement. 


A  number  of  metrics  support  the  appraisal  team  leader’s  monitoring  of  this 
work: 

•  calendar  time  between  initial  contact  and  finalization  of  requirements 

•  effort  expended  to  gather  and  analyze  requirements 

•  number  of  meetings  with  representatives  of  the  sponsoring  and/or 
appraised  organization 


The  exit  criterion  for  this  activity  is  the  formal  approval  of  the  appraisal  input 
and  its  placement  under  change  management. 

Review  of  the  documented  agreements  resulting  from  the  work  of  this  set  of 
activities  will  serve  to  validate  the  requirements,  which  feed  into  appraisal 
planning. 


The  appraisal  input 


The  experience  of  the  sponsor  with  process  appraisals  will  drive  tailoring 
choices  for  this  process. 

•  A  relatively  inexperienced  appraisal  sponsor  will  need  a  great  deal  of 
information  and  collaborative  consultation  to  provide  meaningful  and 
complete  requirements  for  the  appraisal. 

•  Experienced  sponsors  may  have  overly  aggressive  requirements. 


Continued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


This  process  is  a  foundation  for  the  success  or  failure  of  the  entire  appraisal;  it 
is  at  this  point  in  the  appraisal  that  the  most  leverage  exists  for  avoiding 
problems  and  issues  downstream.  Gathering  and  understanding  the 
requirements  for  the  conduct  of  a  SCAMPI  appraisal  is  vital  to  making 
appropriate  decisions  and  providing  value  to  the  sponsor.  Many  examples  of 
problems  encountered  during  appraisals  can  be  traced  to  shortcomings  in  the 
conduct  of  this  process.  The  extent  to  which  the  activities  described  here  are 
distinct  from  the  activities  described  in  the  next  process,  Develop  Appraisal 
Plan,  will  depend  on  the  strategy  and  preferences  of  both  the  appraisal  team 
leader  and  the  appraisal  sponsor. 


The  objectives  that  motivate  the  conduct  of  an  appraisal  must  be  well 
understood  so  that  appropriate  participants,  tailoring  decisions,  and  appraisal 
outputs  can  be  selected.  The  constraints  that  shape  the  appraisal  enactment,  in 
light  of  the  objectives,  may  limit  achievement  of  the  desired  result  if  they  are 
not  adequately  understood  and  negotiated.  A  clear  agreement  regarding 
appraisal  outputs  and  their  intended  usage  will  help  maintain  the  sponsorship 
needed  for  conducting  the  appraisal  and  acting  on  the  results.  Establishing 
agreement  on  these  objectives,  constraints,  outputs,  and  intended  usage  forms 
the  basis  for  a  commitment  to  the  plan  for  conducting  the  appraisal. 


♦ 
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1.1.1  Determine  Appraisal  Objectives 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  business  needs  for  process  improvement  drive  the  requirements  for  the 
conduct  of  any  given  appraisal,  and  generally  include  one  or  more  of  three 
closely  related  factors: 

•  reducing  costs 

•  improving  quality 

•  decreasing  time  to  market 

The  fundamental  premise  of  process  improvement  is  that  organizational 
processes  significantly  impact  these  factors. 

A  fair  and  objective  characterization  of  the  processes  in  use  in  the 
organization(s)  is  the  essential  reason  for  conducting  an  appraisal.  In  addition 
to  this  motivation,  a  sponsor’s  desire  to  conduct  an  appraisal  could  be  driven 
by  one  or  more  of  the  following  business-related  objectives: 

•  Document  a  credible  benchmark  that  reflects  successful  process 
improvement. 

•  Evaluate  areas  of  potential  risk  that  may  affect  the  performance  of  the 
organization. 

•  Involve  members  of  the  appraised  organization  in  improving  the 
performance  of  the  process. 

•  Support  specific  decisions  related  to  the  direction  of  a  new  or  existing 
improvement  program. 

•  Motivate  a  supplier  to  focus  on  process  issues  that  affect  contract 
performance. 


•  Identify  sponsor  and  relevant  stakeholders,  and  establish  communication. 

•  Document  business  and  appraisal  objectives. 

•  Ensure  the  alignment  of  appraisal  objectives  to  business  objectives. 

•  Determine  and  document  the  appraisal  usage  mode  (Internal  Process 
Improvement,  Supplier  Selection,  Process  Monitoring). 


At  least  one  communication  between  the  appraisal  team  leader  and  sponsor  is 
required.  (Some  usage  modes  may  limit  this  significantly;  others  may  require 
more  than  one  interaction.) 


Continued  on  next  page 
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1 .1 .1  Determine  Appraisal  Objectives  (continued) 


Optional  None. 

Practices 


Implementation  Organizations  with  experience  in  the  use  of  appraisals  may  have  a  clear  set  of 
Guidance  appraisal  objectives  identified  in  advance  of  contacting  a  Lead  Appraiser. 

This  provides  the  Lead  Appraiser  with  a  starting  point,  but  does  not  permit 
him  or  her  to  “skip”  this  activity. 

In  some  cases  the  usage  mode  will  be  self-evident;  however,  there  may  be 
instances  where  the  appraisal  sponsor  either  may  not  be  sure  or  may  have 
made  an  assumption  that  is  not  founded  on  fact.  The  appraisal  team  leader  is 
responsible  for  ensuring  that  the  best  choice  of  usage  mode  is  made  consistent 
with  the  sponsor’s  input  and  direction. 

Also  note  that  the  roles  of  appraisal  sponsor  and  senior  site  manager  may  be 
played  by  the  same  person  or  by  two  individuals,  depending  on  the  usage 
mode. 

Depending  on  the  structure  of  the  appraised  organization,  as  well  as  the  usage 
mode,  it  is  often  important  to  distinguish  the  role  of  senior  site  manager  from 
that  of  appraisal  sponsor.  For  some  appraisals,  these  two  roles  are 
encompassed  in  the  duties  of  a  single  person.  For  other  appraisals,  these  two 
roles  may  represent  two  people  working  many  time  zones  away  from  each 
other. 


♦ 
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1.1.2  Determine  Appraisal  Constraints 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  constraints  within  which  the  appraisal  must  be  conducted  are  determined 
based  on  a  dialog  between  the  appraisal  team  leader  and  the  appraisal  sponsor 
and/or  senior  site  manager.  This  typically  is  an  iterative  process  in  which  the 
preferences  of  the  appraisal  sponsor,  the  limits  of  the  method,  and  the 
consequent  resource  requirements  are  balanced  against  each  other  to  arrive  at 
an  optimal  set  of  appraisal  input  parameters. 


•  Establish  high-level  cost  and  schedule  constraints. 

•  Determine  which  process  areas  (PAs)  and  organizational  entities  are  to  be 
included. 

•  Determine  minimum,  maximum,  or  specific  sample  size  or  coverage  that 
is  desired  for  the  appraisal. 

•  Negotiate  constraints  and  objectives  with  stakeholders  to  ensure 
feasibility. 

•  Document  negotiated  constraints  to  be  met. 


At  least  one  communication  between  the  appraisal  team  leader  and  sponsor  is 
required.  (Some  usage  modes  may  limit  this  significantly;  others  may  require 
more  than  one  interaction.) 

Constraints  on  cost  and  schedule  identified  during  this  early  stage  of  the 
appraisal  are  expected  to  be  high-level,  and  not  detailed  estimates.  They  may 
take  the  form  of  statements  such  as  “We  need  this  done  in  Q4,”  “You  can’t 
use  more  than  five  of  my  people  on  the  team,”  and  “I  can’t  afford  to  have  it 
last  more  than  a  month.”  Constraints  identified  by  the  appraisal  input  must  be 
negotiated  between  the  sponsor  and  the  appraisal  team  leader. 


Continued  on  next  page 
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1 .1 .2  Determine  Appraisal  Constraints  (continued) 


Optional  Document  the  rationale  for  choices  made  and  the  associated  tradeoffs  as  a 

Practices  resource  for  later  planning  and  future  appraisals. 


Implementation  Practical  limitations  relating  to  time,  cost,  and  effort  are  clarified  and 
Guidance  negotiated  in  the  context  of  other  requirements  the  sponsor  has.  The  business 

context  in  which  the  appraisal  is  conducted  drives  choices  that  the  appraisal 
team  leader  needs  to  make.  Appraisals  should  not  be  conducted  in  isolation 
from  other  activities  relating  to  process  management  and  improvement.  The 
needs  of  related  stakeholders,  be  they  acquisition  organizations  or  division 
heads  managing  the  Engineering  Process  Group,  often  place  requirements  on 
the  conduct  of  the  appraisal. 


♦ 
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1 .1 .3  Determine  Appraisal  Scope 


Activity 

Description 


Required 

Practices 


The  appraisal  scope  consists  of  the  reference  model  scope  and  the 
organizational  scope  to  be  examined  during  the  appraisal.  The  model  scope 
must  be  determined  and  documented  early  in  the  planning  process,  using  the 
staged  representation  or  the  continuous  representation.  The  appraisal  team 
leader  is  responsible  for  ensuring  that  the  sponsor  makes  an  informed  choice 
regarding  the  PAs  included  in  the  scope  of  the  appraisal  and  the  model 
representation.  The  selection  of  appraisal  outputs  should  be  driven  by  the 
understanding  of  their  intended  use,  established  during  the  requirements 
analysis  activity,  and  may  dictate  some  selections  in  model  scope.  The 
organizational  scope  defines  the  bounds  of  the  organization  to  be  investigated 
during  the  appraisal.  Instantiations  (i.e.,  for  practices  implemented  by 
projects,  each  project;  for  practices  implemented  organization-wide,  the 
instance)  are  selected  as  representative  of  the  organization  and  the  contexts  in 
which  processes  are  implemented. 

Reconciling  the  interactions  between  model  scope  and  organization  scope  is 
an  important  part  of  this  activity.  A  particular  organization  scope  begets  a 
particular  model  scope;  a  particular  model  scope  requires  a  particular 
organization  scope. 


•  Determine  and  document  the  reference  model  scope  and  representation  to 
be  used  for  the  appraisal. 

•  Determine  and  document  the  organizational  unit  to  be  investigated  during 
the  appraisal. 


Continued  on  next  page 
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1 .1 .3  Determine  Appraisal  Scope  (continued) 


Parameters 
and  Limits 


Optional 

Practices 


The  reference  model  scope  includes  the  PAs  and  associated  maximum 
capability  level  and/or  maturity  level  that  will  be  investigated  by  the  appraisal 
team  (i.e.,  the  generic  goals  that  will  be  rated  for  each  PA  within  the  scope  of 
the  appraisal).  Note  that  the  selection  of  the  reference  model  representation 
should  have  been  discussed  during  the  setting  of  appraisal  objectives,  because 
the  representation  selected  may  impact  the  achievability  of  these  objectives. 

The  model  scope  of  the  appraisal  must  encompass  at  least  one  PA.  All  generic 
goals  and  specific  goals  up  to  and  including  the  target  capability  level  and/or 
maturity  level  for  each  selected  PA  must  be  rated;  individual  goals  within  a 
PA  cannot  be  excluded. 

Instantiations  must  be  selected  that  are  representative  of  the  implemented 
processes  and  functional  areas  being  investigated  within  the  organizational 
unit,  and  that  operate  within  a  coherent  process  context  (see  glossary  for 
definition).  This  is  also  sometimes  known  as  the  organizational  scope  of  the 
appraisal.  The  rationale  for  selecting  these  elements  as  representative  of  the 
organizational  unit  should  be  documented. 

Typically,  the  organizational  unit  will  be  specified  in  such  a  manner  that  (a)  at 
least  two  instances  of  the  processes  being  investigated  are  available  as  sources 
of  objective  evidence  and  (b)  a  representative  coverage  of  the  life  cycles  in 
use  within  the  organization  is  obtained.  Selection  of  instantiations  within  the 
organizational  unit  may  be  accomplished  through  a  survey  form,  or  through 
summarizing  information  learned  from  discussions  with  members  of  the 
organization.  For  processes  enacted  at  the  organization  level  (such  as 
Organizational  Training),  multiple  instances  are  not  required. 

The  representative  instantiations  to  be  investigated  during  the  appraisal  will 
also  drive  the  selection  of  participants  needed  to  provide  sources  of  objective 
evidence.  An  initial  determination  of  appraisal  participants,  by  name  and  role, 
should  be  negotiated  with  the  appraisal  sponsor  and/or  the  senior  site  manager 
as  part  of  the  early  determination  of  organizational  scope.  This  will  be  refined 
later  during  detailed  appraisal  planning. 


Use  broad-based  survey  instruments  or  a  Practice  Implementation  Indicator 
(PII)  database  to  characterize  the  population  of  projects  or  divisions  in  an 
organization  before  determining  the  organizational  scope  of  the  appraisal. 


Continued  on  next  page 
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1 .1 .3  Determine  Appraisal  Scope  (continued) 


Implementation 

Guidance 


There  are  two  primary  parameters  of  the  appraisal  enactment  that  contribute 
significantly  to  the  resulting  cost  (in  terms  of  effort):  the  PA  scope  of  the 
CMMI  model  encompassed  by  the  appraisal  and  the  number  and  size  of 
projects  selected.  While  other  parameters  contribute  to  the  cost  and  schedule 
of  an  appraisal,  these  two  scope  parameters  provide  the  greatest  opportunity 
to  shape  the  magnitude  of  the  appraisal.  SCAMPI  requires  that  findings  for 
the  organizational  unit  be  derived  from  objective  evidence  on  the 
implementation  of  practices  collected  from  each  of  the  organizational  process 
instantiations  included  in  the  appraisal.  The  size  and  number  of  instantiations 
investigated  should  be  selected  to  form  a  valid  sample  of  the  organizational 
unit  to  which  the  results  will  be  attributed. 

Clearly,  a  broadly  defined  organizational  unit  (e.g.,  a  multi-national 
enterprise)  will  require  collecting  and  analyzing  significantly  more  objective 
evidence  than  a  narrowly  defined  organizational  unit  (e.g.,  a  specific  product 
line  within  a  specific  business  unit  at  a  single  geographical  location). 

The  organization  to  which  appraisal  results  will  be  attributed  should  be 
described  accurately  in  all  statements  made  by  the  appraisal  team  leader  and 
sponsor.  It  is  the  responsibility  of  the  appraisal  team  leader  to  understand  the 
larger  organizational  context  in  which  the  appraised  organizational  unit 
resides.  Familiarity  with  the  nature  of  departmental  structures,  matrixed 
subject  matter  expert  groups,  integrated  product  teams,  program  and  project 
groupings,  or  product  line  implications  that  may  affect  the  interpretation  of 
appraisal  outcomes  will  aid  in  obtaining  this  understanding. 

The  appraisal  team  leader  should  work  with  representatives  from  the 
organization  to  document  a  clear  statement  of  the  model  and  organizational 
scope  of  the  appraisal.  The  model  scope  should  be  documented  using  a  list  of 
PAs  to  be  included  in  the  appraisal,  as  well  as  the  model  components  to  be 
rated  by  the  appraisal  team.  The  organizational  scope  of  the  appraisal  should 
be  documented  in  the  clearest  terms  possible,  given  the  nature  of  the 
organizational  structure  in  place.  It  is  often  difficult  to  specify  unambiguous 
boundaries  without  resorting  to  naming  individual  people  in  some 
organizations.  Information  about  the  organizational  unit  should  be 
documented  in  a  way  that  allows  future  appraisal  sponsors  to  replicate 
(exactly)  the  scope  of  the  organization  appraised.  This  information  should  be 
in  the  appraisal  plan,  and  used  (in  summary  form  if  needed)  in  briefing  the 
appraisal  team  and  appraisal  participants. 


♦ 
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1.1.4  Determine  Outputs 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Identify  the  specific  appraisal  outputs  to  be  produced.  Some  appraisal  outputs 
are  required  and  additional  outputs  are  tailorable  (see  Parameters  and  Limits 
and  Optional  Practices). 

Obtain  unambiguous  answers  to  the  following  questions: 

•  What  ratings  will  be  generated  during  the  appraisal? 

•  Will  a  final  report  be  written  to  document  appraisal  results? 

•  Will  recommendations  on  how  to  address  specific  findings  be  generated 
and  reported? 


•  Review  required  SCAMPI  outputs  with  the  appraisal  sponsor.  Review  and 
select  optional  SCAMPI  outputs  with  the  appraisal  sponsor. 

•  Determine  the  recipients  of  appraisal  outputs. 


Required  SCAMPI  outputs  include 

•  Appraisal  Record  (see  activity  3.2.2,  Generate  Appraisal  Record) 

•  Appraisal  Disclosure  Statement  (see  activity  2.4.4,  Document  Appraisal 
Results) 

•  CMMI  Steward  Data  (see  activity  3.2.3,  Provide  Appraisal  Feedback  to 
CMMI  Steward) 

As  stated  in  the  ARC,  at  least  all  the  goals  for  the  process  area  or  areas 
investigated  by  the  team  must  be  rated,  although  the  choice  may  be  made  to 
not  disclose  the  ratings  to  anyone  other  than  the  appraisal  sponsor. 

At  a  minimum,  the  sponsor  gets  the  following  appraisal  outputs: 

•  final  findings,  including  statements  of  strengths  and  weaknesses 
documented  by  the  team  for  every  PA  investigated 

•  all  ratings  planned  for  and  generated  by  the  team 

Decisions  reached  on  appraisal  outputs,  including  what  ratings  will  be 
reported,  are  documented  in  the  appraisal  input. 


Continued  on  next  page 
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1 .1 .4  Determine  Outputs  (continued) 


Optional  The  appraisal  sponsor  may  request  that  additional  rating  outputs  be  generated 

Practices  as  a  result  of  the  appraisal.  Typical  rating  outputs  that  might  be  selected 

include 

•  maturity  level  and/or  capability  level  ratings 

•  PA  Satisfaction/Capability  Level  Profiles 

•  practice  ratings 

•  an  option  to  use  “partially  satisfied”  as  a  rating  assigned  to  a  PA 

•  15504  Process  Profile 

•  discipline-specific  ratings  (e.g.,  SE  or  SW) 

•  project-level  findings  or  ratings 

•  other  (non-typical)  outputs  desired 

Many  of  these  optional  appraisal  outputs  are  discussed  further  in  process  2.4, 
Generate  Appraisal  Results. 

The  sponsor  may  also  request  that  other  products  be  generated  as  appraisal 
outputs.  Typical  products  that  might  be  requested  include  (see  activity  3.1.3, 
Plan  for  Next  Steps): 

•  Appraisal  Final  Report 

•  Recommendations  for  taking  action  upon  the  appraisal  results 

•  Process  improvement  action  plan 

Implementation  Goal  satisfaction  ratings  for  both  specific  goals  and  generic  goals  of  the  PAs 
Guidance  within  the  scope  of  the  appraisal  are  a  minimum  requirement.  Capability 

and/or  maturity  level  ratings  are  optional.  There  is  no  requirement  to  report 
the  ratings  to  the  appraisal  participants  even  though  ratings  are  performed. 
The  sponsor  has  sole  authority  to  decide  (in  advance)  what  ratings  will  or  will 
not  be  reported,  and  to  whom  they  will  be  reported. 

While  statements  of  findings  are  a  required  output  of  the  method,  creating  a 
written  report  that  elaborates  on  the  findings  is  optional.  The  sponsor  should 
decide  if  resources  are  to  be  spent  creating  this  artifact.  Similarly,  the  task  of 
creating  recommendations  to  address  issues  uncovered  in  the  appraisal  may 
require  expertise  that  is  not  represented  on  the  appraisal  team  in  some  cases. 
The  characteristics  of  the  appraised  organization  and  the  constraints  that 
shape  its  improvement  program  should  be  carefully  considered  when  making 
process  improvement  recommendations. 
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1 .1 .5  Obtain  Commitment  to  Appraisal  Input 


Activity 

Description 

Required 

Practices 


Parameters 
and  Limits 


The  appraisal  sponsor  formally  approves  the  appraisal  input,  and  this  set  of 
information  is  placed  under  change  management. 


•  Record  required  information  in  the  appraisal  input  record. 

•  Obtain  sponsor  approval  of  the  appraisal  input  record. 

•  Manage  changes  to  the  appraisal  input,  obtaining  sponsor  approval  of 
changes. 


The  appraisal  input  may  be  generated  incrementally  throughout  planning,  but 
must  be  approved  prior  to  the  start  of  data  collection.  At  a  minimum,  the 
appraisal  input  shall  provide  the  information  needed  to  address  the  following: 

•  the  identity  of  the  appraisal  sponsor  and  the  relationship  of  the  sponsor  to 
the  organizational  unit  being  appraised 

•  the  appraisal  purpose,  including  alignment  with  business  objectives  (see 
activity  1.1.1) 

•  the  appraisal  reference  model  scope  (see  activity  1.1.3) 

•  the  organizational  unit  being  appraised  (see  activity  1.1.3) 

•  the  process  context,  which,  at  a  minimum,  includes 

organizational  unit  size  and  demographics 
application  domain,  size,  criticality,  and  complexity 
high-priority  characteristics  (e.g.,  time  to  market,  feature  richness, 
reliability)  of  the  products  and  services  of  the  organizational  unit 

•  appraisal  constraints  (see  activity  1 . 1 .2),  which,  at  a  minimum,  include 

availability  of  key  resources  (e.g.,  staffing,  funding,  tools,  facilities) 
schedule  constraints 

the  maximum  amount  of  time  to  be  used  for  the  appraisal 
specific  PAs  or  organizational  entities  to  be  excluded  from  the 
appraisal 

the  maximum,  minimum,  or  specific  sample  size  or  coverage  desired 
for  the  appraisal 

ownership  of  appraisal  results  and  any  restrictions  on  their  use 
controls  on  information  resulting  from  a  confidentiality  agreement 
non-attribution  of  appraisal  data  to  associated  sources 

•  the  identity  of  the  CMMI  models  used  (version,  discipline,  and 
representation) 
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1 .1 .5  Obtain  Commitment  to  Appraisal  Input  (continued) 


Parameters 
and  Limits 
(continued) 


Optional 

Practices 

Implementation 

Guidance 


•  the  identity  and  affiliation  of  the  Lead  Appraiser  who  is  to  be  the 
appraisal  team  leader  for  the  appraisal 

•  the  identity  and  affiliation  of  the  appraisal  team  members,  with  their 
specific  appraisal  responsibilities 

•  the  identity  (name  and  organizational  affiliation)  of  appraisal  participants 
and  support  staff,  and  their  specific  responsibilities  for  the  appraisal 

•  any  additional  information  to  be  collected  during  the  appraisal  to  support 
the  achievement  of  the  appraisal  objectives 

•  a  description  of  the  planned  appraisal  outputs  (see  activity  1.1.4), 
including  ratings  to  be  generated 

•  anticipated  follow-on  activities  (e.g.,  reports,  appraisal  action  plans,  re¬ 
appraisal) 

•  planned  tailoring  of  SCAMPI  and  associated  tradeoffs,  including  the 
sample  size  or  coverage  of  the  organizational  unit 

•  appraisal  usage  mode  (i.e..  Internal  Process  Improvement,  Supplier 
Selection,  Process  Monitoring) 


None. 


A  Lead  Appraiser’s  ability  to  build  and  maintain  commitment  from  the 
sponsor  and  the  members  of  the  sponsoring  organization  is  a  major  factor 
contributing  to  the  success  of  the  appraisal.  The  process  of  understanding  the 
requirements  and  constraints  should  yield  a  series  of  agreements  that  form  an 
input  to  the  appraisal  plan.  Based  on  the  judgment  of  the  appraisal  team 
leader,  these  agreements  may  be  covered  in  a  formal  (signed)  document  that 
forms  a  basis  for  future  activities.  More  typically,  the  appraisal  team  leader 
maintains  a  record  of  interactions  with  the  sponsor,  which  are  incorporated 
into  the  appraisal  plan  as  it  is  drafted. 

The  appraisal  team  leader  and  the  sponsor  should  have  verbal  agreement  on 
the  items  discussed  above,  and  these  items  should  be  documented  in  some 
way.  The  formality  of  the  documentation  may  range  from  simple  meeting 
minutes  maintained  by  the  appraisal  team  leader,  to  a  more  formal 
Memorandum  of  Understanding  or  other  vehicle  that  documents  agreements 
and  provides  traceability.  It  is  expected  that  the  appraisal  plan  will  be  used  to 
document  important  issues  pertaining  to  requirements. 
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1 .2  Develop  Appraisal  Plan 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Document  the  results  of  appraisal  planning  including  the  requirements, 
agreements,  estimates,  risks,  method  tailoring,  and  practical  considerations 
(e.g.,  schedules,  logistics,  and  contextual  information  about  the  organization) 
associated  with  the  appraisal.  Obtain  and  record  the  sponsor’s  approval  of  the 
appraisal  plan. 


An  appraisal  sponsor  and  SCAMPI  Lead  Appraiser  have  agreed  to  proceed 
with  appraisal  planning,  based  on  a  common  understanding  of  the  key 
parameters  that  drive  the  planning  process. 


Documented  agreement(s),  reflected  in  the  appraisal  input,  that  support  a 
common  understanding  of  appraisal  objectives  and  key  appraisal-planning 
parameters. 


1.2.1  Tailor  Method 

1.2.2  Identify  Needed  Resources 

1.2.3  Determine  Cost  and  Schedule 

1.2.4  Plan  and  Manage  Logistics 

1.2.5  Document  and  Manage  Risks 

1.2.6  Obtain  Commitment  to  Appraisal  Plan 


•  Approved  appraisal  plan 

•  Strategy  for  managing  logistics 

•  Strategy  for  preparing  the  organization(s) 

•  Schedule 

•  Interview  plan 

•  Team  assignments 


The  sponsor  and  appraisal  team  leader  agree  on  technical  and  non-technical 
details  for  the  planned  appraisal.  The  plan  is  refined  in  conjunction  with 
performing  the  other  Planning  and  Preparation  phase  activities.  This  agreement 
is  documented  and  reviewed  by  affected  stakeholders  as  appropriate. 


The  final  appraisal  plan  is  reviewed  and  approved. 
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1.2  Develop  Appraisal  Plan  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 

Verification  and 
Validation 

Records 

Tailoring 


Skilled  appraisal  team  leaders  will  effectively  develop  and  use  outputs  from 
the  other  Planning  and  Preparation  phase  activities  to  achieve  clarity  of  the 
shared  vision  necessary  to  make  the  tradeoffs  and  decisions  resulting  in  a  final 
plan.  This  activity  is  an  important  opportunity  for  the  appraisal  team  leader  to 
demonstrate  process  discipline,  as  well  as  the  type  of  careful  planning 
described  in  the  CMMI  model.  Experienced  appraisal  team  leaders  will 
leverage  data,  templates,  and  assets  (developed  through  their  own  experience) 
to  improve  the  completeness  and  effectiveness  of  the  appraisal  plan, 
recognizing  the  return  on  investment  that  will  be  obtained  through  smooth  and 
efficient  appraisals. 


Tools  include  an  appraisal  plan  template,  samples,  and  embedded  procedural 
guidance  in  planning  templates.  Estimation  worksheets  and  methods  for 
assessing  the  impact  of  appraisal  constraints  are  also  quite  useful. 


•  Calendar  time  spanned  by  the  activity 

•  Effort  consumed  in  carrying  out  the  activities  of  this  process 

•  Level  and  frequency  of  changes  to  the  appraisal  plan 


•  Comparison  of  actual  effort  for  this  activity  with  historical  data 
accumulated  by  the  appraisal  team  leader 

•  Review  of  the  appraisal  plan  by  affected  stakeholders 

•  Sponsor’s  approval  of  the  plan 


•  Estimation  worksheets  (if  used) 

•  Appraisal  plan  (see  activity  1.2.6  for  a  detailed  list  of  plan  contents) 


•  In  some  applications,  planning  templates  and  procedures  in  routine  use 
within  the  organization  can  be  adapted  to  the  needs  of  the  appraisal.  This 
aids  in  communication  as  well  as  local  ownership  of  the  process. 

•  A  structured  planning  workshop  may  be  of  benefit  for  organizations  with 
limited  appraisal  experience.  Such  a  workshop  is  a  valuable  opportunity 
to  discover  risks  as  well  as  mitigation  strategies. 
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1.2  Develop  Appraisal  Plan  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  appraisal  plan  will  guide  and  define  the  execution  of  the  appraisal  such 
that  it  is  in  concert  with  the  business  needs  and  constraints.  An  initial  plan  can 
be  generated  immediately  following  consultation  with  the  sponsor.  Further 
refinement  is  done  as  detailed  planning  occurs  and  new  information  comes  to 
light  in  executing  appraisal  plantiing  and  preparation.  A  final  appraisal  plan 
must  be  completed  prior  to  the  completion  of  process  1.5,  Prepare  for 
Collection  of  Objective  Evidence.  Typically,  resources,  method  tailoring, 
model-related  decisions,  and  a  planned  list  of  outputs  are  finalized  early  on, 
while  cost,  schedule,  and  logistics  are  finalized  later  in  the  Plan  and  Prepare 
for  Appraisal  phase. 

The  appraisal  input  is  a  necessary  input  to  the  appraisal-planning  process. 
While  it  may  not  be  necessary  to  formally  separate  the  requirements  analysis 
activities  from  the  activities  described  in  this  section,  prior  understanding  of 
the  appraisal  requirements  is  a  necessary  input  to  this  process.  The  plan  for 
the  appraisal  provides  an  important  vehicle  for 

•  documenting  agreements  and  assumptions 

•  establishing  and  maintaining  sponsorship 

•  tracking  and  reporting  the  performance  of  the  appraisal  process 

•  reinforcing  commitments  at  key  points  in  the  appraisal  process 

The  distinction  between  the  appraisal  input  and  appraisal  plan  is  intended  to 
separate  the  key  appraisal  requirements  and  strategic  objectives,  which  require 
high  sponsor  visibility  and  change  control  approval,  from  the  tactical  planning 
details  necessary  to  implement  and  satisfy  these  objectives.  While  sponsor 
visibility  into  the  appraisal  plan  is  necessary,  revisions  are  typically  low-level 
implementation  details  and  do  not  ordinarily  require  sponsor  re-approval.  In 
practical  use,  the  appraisal  input  is  often  packaged  as  a  component  of  the 
appraisal  plan,  and  a  single  sponsor  signature  can  serve  as  approval  for  both. 


This  process  is  composed  of  six  activities  summarized  here  and  described 
below.  The  scope  of  the  appraisal  is  defined  in  terms  of  (a)  the  portion  of  the 
CMMI  model  that  will  be  investigated  and  (b)  the  bounds  of  the 
organizational  unit  for  which  the  results  can  be  considered  valid  (e.g.,  a 
project,  a  product  line,  an  operating  division,  a  business  unit,  an  entire  global 
enterprise).  Method-tailoring  choices  are  made  to  most  effectively  achieve 
appraisal  objectives  within  defined  constraints  of  time,  effort,  and  cost.  The 
resources  required  to  carry  out  the  appraisal  are  identified.  The  cost  and 
schedule  are  negotiated  and  recorded.  The  details  of  logistics,  particularly  for 
the  on-site  period,  are  documented.  Risks  and  risk-mitigation  plans  are 
identified  and  documented.  Completion  of  these  activities  results  in  a  well- 
defined,  achievable  appraisal  plan. 
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1.2.1  Tailor  Method 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Tailoring  of  SCAMPI  includes 

•  selection  of  choices  (if  any)  within  the  Required  Practices 

•  setting  parameters  that  are  allowed  to  vary  within  the  Parameters  and 
Limits 

•  inclusion  of  Optional  Practices 

Because  SCAMPI  is  designed  to  apply  to  a  wide  range  of  appraisal 
applications,  the  tailoring  activity  is  one  that  deserves  careful  and  thoughtful 
attention. 

Using  “partially  satisfied”  and  choosing  to  do  the  appraisal  in  “verification” 
or  “discovery”  mode  are  explicit  selectable  tailoring  options.  This  document 
is  designed  to  clearly  indicate  which  aspects  of  the  method  are  required  and 
which  are  tailorable.  The  Parameters  and  Limits  and  Optional  Practices 
sections  of  each  activity  description  provide  discussions  of  tailoring  options, 
in  context. 

In  addition,  the  appraisal  usage  mode  will  determine  some  tailoring  choices. 


•  Review  and  select  tailoring  options  within  the  Required  Practices  in  each 
activity. 

•  Review  and  set  parameters  within  acceptable  limits,  where  variation  is 
expected. 

•  Review  and  select  appropriate  Optional  Practices. 

•  Ensure  that  the  tailoring  decisions  are  self-consistent  and  that  they  are 
appropriate  in  light  of  the  appraisal  objectives  and  constraints. 

•  Document  the  tailoring  decisions  made. 


The  structure  of  the  MDD  clarifies  which  SCAMPI  features  are  required, 
either  as  a  direct  derivative  of  ARC  requirements  or  as  SCAMPI 
requirements.  Parameters  and  Limits  sections  define  the  allowable  variation 
within  these  method  requirements.  Tailoring  guidance  and  Implementation 
Guidance  are  provided  to  assist  with  tuning  the  method  to  fit  sponsor 
objectives  and  appraisal  constraints.  Method  tailoring  and  implementation 
options  must  be  selected  and  implemented  in  a  way  that  does  not  violate 
SCAMPI  Required  Practices. 
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1.2.1  Tailor  Method  (continued) 


Optional  Provide  the  sponsor  with  more  than  one  candidate  scenario  for  the  appraisal. 

Practices  and  help  them  select  among  the  options. 

Alternatively,  the  appraisal  team  leader  may  define  a  tailored  instance  of  the 
method  and  propose  it  to  the  sponsor  for  approval  or  negotiate  some  of  the 
details. 


Implementation  This  appraisal  method  offers  a  wide  variety  of  choices  that  allow  the  appraisal 
Guidance  team  leader  and  sponsor  to  select  appraisal  features  that  best  address  appraisal 

and  business  objectives.  The  SCAMPI  Implementation  Model  is  an  asset 
provided  to  Lead  Appraisers  by  the  CMMI  Steward  that  assists  with 
understanding  SCAMPI  tailoring  and  implementation  choices. 

Method  tailoring  is  directly  related  to  the  organizational  scope  and  model 
scope  decisions.  Most  of  the  allowable  tailoring  options  flow  logically  from 
these  decisions  when  taken  in  context  of  the  appraisal  objectives  and 
constraints.  Tailoring  decisions  typically  affect  the  appraisal  risk.  Typical 
tailoring  choices  that  significantly  impact  appraisal  planning  include 

•  CMMI  model  PAs  encompassed  by  the  appraisal 

•  specification  of  the  organizational  unit  to  be  appraised 

•  number,  experience,  skills,  and  affiliation  (e.g.,  intemal/extemal)  of 
appraisal  team  members 

•  data  collection,  analysis,  and  validation  approaches  to  be  utilized 
including  supporting  work  aids  and  tools 

•  effort  invested  by  the  organization  and  the  appraisal  team  in  preparation, 
including  pre-on-site  data  collection  and  analysis 

•  time  spent  on  site 

Experienced  appraisal  team  leaders  will  provide  a  well-defined  approach  to 
ensure  that  the  appraisal  objectives  are  achieved  in  an  efficient  and  effective 
manner.  Experienced  sponsors  will  require  a  well-defined  approach  to  ensure 
an  acceptable  level  of  risk  in  meeting  objectives  within  the  constraints.  The 
appraisal  plan  documents  the  method-tailoring  decisions  and  their  rationale, 
and  the  associated  method  variations  and  techniques  that  will  be  employed. 


♦ 


CMU/SEI-2001-HB-001 


Page  11-23 


1 .2.2  Identify  Needed  Resources 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


This  activity  is  concerned  with  the  identification  and  estimation  of  resources 
needed  to  carry  out  the  appraisal.  Resources  include  personnel,  facilities, 
tools,  and  access  to  information. 


•  Identify  appraisal  team  members. 

•  Identify  appraisal  participants. 

•  Identify  equipment  and  facilities. 

•  Identify  other  appraisal  resources  needed. 

•  Document  resource  decisions  in  the  appraisal  plan. 


The  level  of  detail  in  the  identification  of  needed  resources  must  be  sufficient 
to  support  the  creation  of  the  appraisal  plan.  For  example,  the  appraisal  team 
leader  must  identify 

•  the  names  of  people  who  are  candidates  for  interviews  or  appraisal  team 
membership 

•  the  organizational  or  project  affiliation  of  these  people 

•  the  location,  seating  capacity,  and  configuration  of  rooms  to  be  used  by 
the  team 

•  specific  equipment  needed  (e.g.,  overhead  projector,  laptop  projector, 
video-conferencing) 


Several  months  before  the  appraisal,  tour  the  facility  where  the  appraisal  will 
be  held. 

Assign  an  individual  from  the  appraised  organization  to  carry  out  the  duties  of 
the  Organizational  Unit  Coordinator  (administrative  and  logistical  support; 
see  activity  1.3.2.) 
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1.2.2  Identify  Needed  Resources  (continued) 


Implementation 

Guidance 


Appraisal  resources  are  typically  defined  early  in  the  appraisal-planning 
process.  Identifying  resources  goes  hand  in  hand  with  estimating  appraisal 
cost  and  schedule  (see  activity  1.2.3),  and  these  may  be  iteratively  refined. 
Tradeoffs  are  routinely  made  in  light  of  the  appraisal  objectives  and 
constraints. 

The  appraisal  sponsor  or  senior  site  manager  may  identify  candidate  appraisal 
team  members  and  appraisal  participants.  Review  of  the  organizational  unit 
structure  or  other  site-specific  information  can  also  be  useful  for  this.  Initially, 
participants  can  be  specified  in  terms  of  roles  or  responsibilities,  with  specific 
names  to  be  determined  later.  Process  1.3  contains  additional  guidance  on 
selecting  appraisal  team  members. 

Equipment  and  facilities  are  often  negotiated  with  the  organizational  unit 
where  the  appraisal  on-site  activities  will  be  performed,  but  sometimes  these 
must  be  acquired.  A  room  for  dedicated  use  by  the  appraisal  team  is  usually 
necessary  for  private  discussions  and  to  protect  the  confidentiality  of  appraisal 
data.  Ideally,  this  is  separate  from  the  other  rooms  where  interview  sessions 
are  held. 

The  availability  of  computing  resources,  such  as  computers,  printers,  and 
networks,  is  a  key  consideration  that  should  be  planned  and  understood. 
Access  to  special  tools  or  applications  may  also  be  needed. 
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1 .2.3  Determine  Cost  and  Schedule 


Activity 

Description 

Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


A  top-level  cost  breakdown  and  schedule  are  developed  and  included  in  the 
plan. 


•  Estimate  the  duration  of  key  events  as  a  basis  for  deriving  a 
comprehensive  schedule. 

•  Estimate  the  effort  required  for  the  people  participating  in  the  appraisal. 

•  Estimate  the  costs  associated  with  using  facilities  and  equipment  (as 
appropriate). 

•  Estimate  the  costs  for  incidentals  (e.g.,  travel,  lodging,  meals)  as 
appropriate. 

•  Document  detailed  schedule  estimates  in  the  appraisal  plan. 

•  Document  detailed  cost  estimates  in  the  appraisal  plan. 


Effort  estimates  should  be  developed  not  only  for  the  appraisal  team,  but  also 
for  the  expected  participants  within  the  organizational  unit  (e.g.,  interviewees, 
respondents  to  instruments  administered,  attendees  at  briefings,  support  staff). 

Scheduling  for  each  day  of  the  appraisal  is  required. 


None. 
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1 .2.3  Determine  Cost  and  Schedule  (continued) 


Implementation 

Guidance 


Cost  and  schedule  may  be  developed  top  down  based  upon  sponsor  objectives 
and  constraints,  bottom  up  based  upon  results  of  other  planning  and 
preparation  processes  and  activities,  or  more  generally  using  a  combination  of 
the  two  approaches.  Scheduling  the  events  and  activities  of  the  appraisal  is  an 
ongoing  logistical  task  that  requires  the  coordination  of  many  different  groups 
of  individuals.  Determining  and  communicating  a  schedule  for  the  appraisal, 
and  maintaining  ongoing  visibility  as  the  details  take  form,  is  the  primary 
responsibility  of  the  appraisal  team  leader.  The  Organizational  Unit 
Coordinator  is  expected  to  provide  support  in  this  task,  and  the  appraisal  team 
leader  typically  selects  the  person  who  plays  that  role  with  this  duty  in  mind. 

The  needs  of  the  sponsor  for  appraisal  outputs  of  a  specified  quality  fulfilling 
a  specified  purpose,  balanced  against  the  resources  available  to  conduct  the 
appraisal,  will  determine  the  schedule  constraints.  Schedule  and  cost  need  to 
be  considered  for  the  entire  span  of  the  appraisal  activities.  The  tradeoff 
between  time  spent  in  preparation  versus  time  spent  on  site  will  therefore  be  a 
significant  factor,  as  will  post-on-site  reporting  activities. 

Organizational  costs  for  preparing  and  supporting  appraisals  can  be  reduced 
by  gathering  and  maintaining  objective  evidence  for  each  project  instance.  In 
addition  to  providing  an  effective  mechanism  for  monitoring  the  process 
implementation  and  improvement  progress  of  each  project,  this  enables  the 
ready  availability  and  reuse  of  objective  evidence  for  subsequent  appraisals. 

While  the  schedule  for  the  appraisal  will  be  shared  with  a  fairly  wide 
audience,  the  cost  of  the  appraisal  (or  elements  within  the  appraisal)  is  often 
kept  from  wide  view,  due  to  the  potentially  sensitive  nature  of  this 
information. 
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1.2.4  Plan  and  Manage  Logistics 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 

Implementation 

Guidance 


The  logistical  details  of  the  on-site  portion  of  the  appraisal  are  negotiated  and 
documented.  The  appraisal  team  leader,  supported  by  the  Organizational  Unit 
Coordinator,  manages  planning  tasks  that  document  and  communicate 
logistical  arrangements.  Checklists  and  action  item  tracking  mechanisms  are 
very  important  structures  used  to  manage  these  tasks. 


•  Document  logistical  schedules  and  dependencies. 

•  Maintain  communication  channels  for  providing  status. 

•  Assign  responsibilities  for  tracking  logistical  issues. 


Effective  planning  depends  on  anticipating  a  variety  of  logistical  issues  that 
may  occur  during  the  appraisal.  Issues  that  are  sometimes  overlooked  include 

•  identifying  hotels  for  people  traveling  to  the  appraisal 

•  providing  workstation  support 

•  ordering  meals 

•  interacting  with  facilities  staff  on  site 

•  meeting  security/classification  requirements 

•  providing  badges  or  arranging  for  escorts  in  limited-access  facilities 


None. 


Every  experienced  appraisal  team  leader  knows  the  value  of  thorough 
logistical  planning  and  tracking.  The  time-critical  nature  of  on-site  appraisal 
activities  makes  it  very  difficult  to  manage  last-minute  changes  in  important 
details  such  as  the  following: 

•  availability  of  conference  rooms  and  meeting  rooms  of  the  appropriate 
size 

•  access  to  rooms,  equipment,  and  supplies  needed  for  administrative  tasks 

•  transportation  and/or  lodging  for  team  members  or  the  remote  members  of 
the  organizational  unit 

•  food  and  other  amenities  required  for  adequate  working  conditions 

•  communication  channels  and  back-up  staff  to  support  the  team  on  site 
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1.2.5  Document  and  Manage  Risks 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 

Impiementation 

Guidance 


As  with  any  project  containing  dependencies  among  events,  people,  and  other 
resources,  risk  management  is  an  important  ingredient  to  success.  The 
appraisal  team  leader  is  responsible  for  documenting  and  communicating  risks 
and  associated  mitigation  plans  to  the  sponsor  and  appraisal  team  members. 


•  Identify  appraisal  risks. 

•  Develop  mitigation  plans  for  key  appraisal  risks,  and  implement  these 
plans  as  necessary. 

•  Keep  the  appraisal  sponsor  and  other  stakeholders  informed  of  the 
appraisal  risk  status. 


The  risks  and  mitigation  plans  identified  through  conducting  this  activity  are 
required  elements  of  the  appraisal  plan  (see  Parameters  and  Limits  for  activity 
1.2.6).  Most  Lead  Appraisers  include  a  section  titled  “Risk  Management”  in 
the  appraisal  plan.  The  level  of  effort  devoted  to  risk-management  activities  is 
something  the  appraisal  team  leader  must  adjust  to  fit  the  situation  at  hand. 


None. 


The  appraisal  plan  is  used  to  document  and  track  risks  to  the  successful 
conduct  of  the  appraisal.  As  with  the  requirement  to  address  logistical  issues 
during  planning,  there  are  no  minimum  guidelines  to  be  met  other  than  the 
requirement  that  the  plan  include  identified  risks  and  planned  mitigation 
strategies. 

The  appraisal  team  leader  is  responsible  for  keeping  the  appraisal  sponsor 
informed  of  risk-management  activities  so  that,  if  needed,  timely  sponsor 
intervention  is  possible  to  ensure  the  achievement  of  appraisal  objectives. 
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1.2.6  Obtain  Commitment  to  Appraisai  Plan 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Formal  sponsor  commitment  is  obtained  to  the  appraisal  plan.  The  appraisal 
plan  constitutes  a  “contract”  between  the  appraisal  sponsor  and  the  appraisal 
team  leader,  so  it  is  vital  that  this  agreement  be  formal. 


•  Document  the  appraisal  plan. 

•  Review  the  appraisal  plan  with  the  sponsor  and  secure  the  sponsor’s 
approval. 

•  Provide  the  appraisal  plan  to  relevant  stakeholders  for  review. 


Required  contents  of  the  appraisal  plan  include  the  following,  at  a  minimum: 

•  the  appraisal  input  (see  activity  1.1.5) 

•  the  activities  to  be  performed  in  conducting  the  appraisal 

•  resources  needed  for  conducting  the  appraisal  (see  1 .2.2) 

•  cost  and  schedule  estimates  for  performing  the  appraisal  (see  activity 
1.2.3) 

•  appraisal  logistics  (see  activity  1.2.4) 

•  risks  and  mitigation  plans  associated  with  appraisal  execution  (see 
activity  1.2.5) 

•  the  criteria  to  verify  that  the  requirements  of  ISO/IEC  15504  have  been 
met,  if  requested  by  the  appraisal  sponsor 

There  must  be  a  signature  block  for  the  appraisal  team  leader  and  the  sponsor 
to  indicate  in  writing  their  commitment  to  the  plan.  If  minor  updates  are  made 
to  the  plan,  signatures  do  not  have  to  be  obtained  again  except  when  one  or 
more  elements  of  the  appraisal  input  have  been  changed. 

At  a  minimum,  the  appraisal  team  members  are  considered  relevant 
stakeholders  and  should  receive  a  copy  of  the  approved  appraisal  plan. 
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1 .2.6  Obtain  Commitment  to  Appraisal  Plan  (continued) 


Optional  Use  a  signature  block  for  relevant  stakeholders  to  indicate  in  writing  their 

Practices  commitment  to  the  plan  (i.e.,  each  team  member  signs  the  plan). 


Implementation  While  sponsor  visibility  into  the  appraisal  plan  is  necessary,  revisions  are 
Guidance  typically  low-level  implementation  details  and  do  not  ordinarily  require 

sponsor  re-approval.  This  is  in  contrast  to  the  appraisal  input,  which  contains 
strategic,  key  appraisal  requirements,  objectives,  and  constraints.  Revisions  to 
the  appraisal  input  must  be  approved  by  the  sponsor.  In  practical  use,  the 
appraisal  input  is  often  packaged  as  a  component  of  the  appraisal  plan,  and  a 
single  sponsor  signature  can  serve  as  approval  for  both.  The  separation  of  the 
appraisal  input  and  appraisal  plan  is  intended  to  provide  an  appropriate  level 
of  sponsor  visibility  and  approval,  while  leaving  appraisal  team  leaders  the 
flexibility  to  refine  the  low-level  details  necessary  to  complete  thorough 
appraisal  planning. 

The  use  of  the  term  “relevant  stakeholder’’  in  the  context  of  appraisal  planning 
is  intended  to  be  interpreted  broadly  to  include  as  many  of  the  participants  and 
other  affected  parties  as  feasible. 
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1 .3  Select  and  Prepare  Team 


Purpose 
Entry  Criteria 

Inputs 

Activities 

Outputs 

Outcome 

Exit  Criteria 


Ensure  that  an  experienced,  trained,  appropriately  qualified  team  is  available 
and  prepared  to  execute  the  appraisal  process. 


•  Appraisal  requirements  have  been  documented  (at  least  in  draft  form). 

•  Appraisal  constraints  are  understood  and  documented  (at  least  in  draft 
form). 

•  The  appraisal  plan  is  defined  (at  least  in  draft  form). 


•  Appraisal  requirements  and  constraints  (in  draft  or  final  form) 

•  Appraisal  plan  (in  draft  or  final  form) 

•  Team  training  materials 


1.3.1  Identify  Team  Leader 

1 .3.2  Select  Team  Members 

1.3.3  Prepare  Team 


•  Training  records 

•  Appraisal  team  member  assignments  and  qualifications 

•  A  prepared  appraisal  team  that  has  completed 

appraisal  method  training 
reference  model  training 
team-building  activities 
team  orientation  regarding  appraisal 


The  successful  completion  of  this  process  results  in  an  experienced,  trained, 
and  oriented  team  ready  to  execute  the  appraisal.  The  appraisal  team  members 
have  acquired  the  necessary  knowledge  to  play  their  roles,  or  their  previous 
knowledge  is  confirmed  to  be  satisfactory.  The  appraisal  team  leader  has 
provided  opportunities  to  practice  the  skills  needed  for  each  person  to  play  his 
or  her  role,  or  has  confirmed  that  these  skills  have  already  been  demonstrated 
in  the  past.  The  team  members  have  been  introduced  to  one  another,  and  have 
begun  to  plan  how  they  will  work  together. 


•  The  prepared  team  is  committed  to  the  appraisal. 

•  Training  has  been  provided  and  its  results  recorded. 

•  Remediation  of  knowledge/skill  shortfalls  has  been  completed  (if 
needed). 


Continued  on  next  page 


CMU/SEI-2001-HB-001 


Page  II-33 


1 .3  Select  and  Prepare  Team  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification  and 
Validation 


Records 


Tailoring 


Whether  the  appraisal  team  leader  trains  an  intact  team  or  forms  a  team  from  a 
corps  of  experienced  team  members,  the  responsibility  to  ensure  that  the  team 
is  ready  to  succeed  rests  with  the  appraisal  team  leader. 


Training  course  material  is  available  from  the  CMMI  Steward  for  training 
teams.  This  should  be  tailored  or  supplemented  by  the  appraisal  team  leader 
based  on  the  appraisal  context  or  degree  of  team  member  experience.  Case 
studies  and  exercises  are  recommended  to  reinforce  the  situations  team 
members  are  likely  to  encounter  during  the  appraisal. 

Other  ways  of  accomplishing  this  activity  may  draw  on  one  or  more  of  the 
following: 

•  providing  supplementary  training  to  previously  experienced  team  members, 
so  that  the  operational  details  of  the  approach  used  will  be  familiar 

•  training  a  cadre  of  team  members  and  keeping  their  knowledge  and  skills 
up-to-date,  as  part  of  an  overall  program  of  appraisals 


Summary  of  team  member  qualifications 

Effort  and  calendar  time  expended  to  accomplish  training 

Trainee  ratings  of  instmctional  materials  and  approach  (if  applicable) 

Achievement  of  milestones  for  remedial  activities  (if  applicable) 


Sponsor  and  appraisal  team  leader  approval  of  team  membership  and 
preparation 

Results  of  exams  used  to  demonstrate  training  effectiveness  (if  used) 
Feedback  from  team  members  on  their  readiness  to  perform  their  role(s) 


Team  member  contact  information 
Training  records  (if  applicable) 

Feedback  provided  by  trainees  (if  applicable) 

Team  qualification  summary  (recorded  in  appraisal  plan) 


Case  study  materials  provide  a  variety  of  options  for  expanding  the  team 
training  course  to  add  emphasis  where  more  is  desired. 

Experienced  appraisal  team  leaders  have  had  success  conducting  role- 
plays  and  simulated  appraisal  activities  without  case  studies  as  well. 

When  assembling  a  team  of  already-trained  members,  it  is  important  to 
conduct  team-building  activities  to  ensure  team  cohesion.  Many  team 
building  exercises  are  available  for  this  purpose 
Team  size,  skills,  and  composition  are  tailoring  options  in  the  method. 
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1 .3  Select  and  Prepare  Team  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


This  process  includes  selecting  and  preparing  the  appraisal  team.  It  may  occur 
after  obtaining  sponsor  commitment  to  the  appraisal  input.  The  appraisal  plan 
should  be  available,  at  least  in  draft  form,  as  a  necessary  input  (see  activity 
1.2.6  for  contents).  Selected  appraisal  team  members  may  provide  input  into 
further  definition  of  the  appraisal  planning.  Appraisal  team  training  may 
provide  an  initial  means  to  obtain  a  preliminary  understanding  of  the 
appraised  organization’s  operations  and  processes.  If  available,  the 
organizational  unit’s  PH  database  is  a  useful  resource  for  orienting  the 
appraisal  team  on  organizational  characteristics,  such  as  the  application 
domain,  the  organizational  structure,  the  process  improvement  structure,  and 
approaches  for  reference  model  implementation. 


The  appraisal  team  is  a  cohesive  unit  of  trained  and  capable  professionals, 
each  of  whom  must  meet  stringent  qualifications.  An  appraisal  team  leader  is 
selected  to  plan  and  manage  the  performance  of  the  appraisal,  delegate 
appraisal  tasks  to  team  members,  and  ensure  adherence  to  SCAMPI 
requirements.  Appraisal  team  members  are  selected  based  on  defined  criteria 
for  experience,  knowledge,  and  skills  to  ensure  an  efficient  team  capable  of 
satisfying  the  appraisal  objectives.  Training  is  provided  to  ensure  proficiency 
in  the  reference  model  and  appraisal  method. 
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1.3.1  identify  Team  Leader 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


The  appraisal  sponsor  is  responsible  for  selecting  an  appraisal  team  leader 
who  has  the  appropriate  experience,  knowledge,  and  skills  to  take 
responsibility  for  and  lead  the  appraisal.  By  definition  an  appraisal  team 
leader  must  be  a  SCAMPI  Lead  Appraiser,  authorized  by  the  SEI  Appraiser 
Program,  and  must  be  a  member  of  that  program  in  good  standing.  The  SEI 
Appraiser  Program  is  described  on  the  SEI  Web  site  at 
<http://www.sei.cmu.edu/managing/app.directory.html>.  The  appraisal  team 
leader  is  responsible  for  ensuring  that  the  appraisal  is  conducted  in  accordance 
with  SCAMPI  requirements,  with  tailoring  to  meet  appraisal  objectives  and 
constraints  within  allowable  bounds  defined  by  the  method. 


•  Select  an  authorized  SCAMPI  Lead  Appraiser  to  serve  as  the  appraisal 
team  leader. 

•  Verify  the  qualifications  of  the  appraisal  team  leader  (experience, 
knowledge,  and  skills). 


The  appraisal  team  leader  must  be  an  SEI-authorized  SCAMPI  Lead 
Appraiser  in  good  standing.  This  can  be  verified  on  the  Web  or  by  contacting 
the  SEI  CMMI  Steward  directly. 

There  can  be  only  one  official  appraisal  team  leader  on  any  given  appraisal. 
The  appraisal  team  leader  has  sole  discretion  to  delegate  important  tasks  to 
appraisal  team  members,  but  cannot  delegate  leadership  responsibility  or 
ultimate  responsibility  for  the  successful  completion  of  the  appraisal.  The 
inclusion  of  multiple  Lead  Appraisers  on  a  team  for  a  given  appraisal  can  be  a 
strong  asset  for  the  leader  of  that  team.  However,  the  single  designated 
appraisal  team  leader  must  perform  the  leadership  role  and  manage  the 
appraisal  process. 


In  some  uses  of  SCAMPI,  representatives  of  the  appraisal  sponsor  may 
perform  a  substantial  part  of  the  appraisal  team  leader’s  responsibilities  in 
advance  of  the  initial  identification  of  an  appraisal  team  leader.  Infrastructures 
established  to  manage  Supplier  Selection,  for  example,  may  employ  standard 
acquisition  processes  that  have  well-understood  interfaces  with  the  appraisal 
process. 
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1 .3.1  Identify  Team  Leader  (continued) 


Implementation 

Guidance 


SCAMPI  Lead  Appraisers,  by  definition,  will  have  participated  on  a 
minimum  of  three  appraisals  (two  as  an  appraisal  team  member  and  one  as  an 
appraisal  team  leader).  These  requirements  are  outlined  in  the  SEI  Lead 
Appraiser  program.  An  additional  consideration  impacting  team  experience 
requirements,  however,  is  the  appraisal  usage  mode  for  SCAMPI.  Additional 
experience  may  be  necessary  for  the  appraisal  team  leader  and/or  appraisal 
team  members  if  the  appraisal  is  for  Supplier  Selection  and/or  Process 
Monitoring  or  if  it  will  focus  heavily  on  one  of  the  other  available  disciplines 
or  environments,  such  as  acquisition  or  Integrated  Product  and  Process 
Development.  Similarly,  if  the  appraisal  will  be  used  in  a  high  maturity 
organization  (maturity  levels  4-5  or  capability  levels  4-5),  special  experience, 
training,  and/or  expertise  (e.g,,  statistical  process  control)  may  be  necessary 
for  that  specific  appraisal. 

Appraisal  team  leader  responsibilities  are  defined  and  described  throughout 
the  SCAMPI  MDD,  but  a  summary  overview  of  these  responsibilities 
includes  the  following: 

•  Confirm  the  sponsor’s  commitment  to  proceed  with  the  appraisal. 

•  Ensure  that  appraisal  participants  are  briefed  on  the  purpose,  scope,  and 
approach  of  the  appraisal. 

•  Ensure  that  all  appraisal  team  members  have  the  appropriate  experience, 
knowledge,  and  sldlls  in  the  appraisal  reference  model  and  in  SCAMPI. 

•  Ensure  that  the  appraisal  is  conducted  in  accordance  with  the  documented 
SCAMPI  method. 

•  Verify  and  document  that  the  appraisal  method  requirements  have  been 
met. 

The  appraisal  team  leader  may  be  selected  at  any  time  in  the  appraisal¬ 
planning  phase;  preferably,  the  appraisal  team  leader  is  selected  upon 
initiation  of  appraisal  activities  so  that  he  or  she  may  participate  in  analyzing 
the  requirements  with  the  appraisal  sponsor.  In  any  event,  the  appraisal  team 
leader  should  be  identified  in  time  to  (a)  review  and  approve  the  appraisal 
plan  with  the  appraisal  sponsor  prior  to  beginning  the  on-site  portion  of  the 
appraisal,  and  (b)  ensure  adequate  planning  and  the  preparation  of  appraisal 
team  members. 
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1 .3.2  Select  Team  Members 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


This  activity  involves  identifying  available  personnel,  assessing  their 
qualifications,  and  selecting  them  to  become  appraisal  team  members.  It  may 
occur  after  obtaining  the  sponsor’s  commitment  to  conduct  the  appraisal  and 
may  provide  input  to  the  appraisal  planning. 


•  Ensure  that  minimum  criteria  for  individual  team  members  are  met. 

•  Ensure  that  minimum  criteria  for  the  team  as  a  whole  are  met. 

•  Document  the  qualifications  and  responsibilities  of  team  members  in  the 
appraisal  input. 


The  minimum  acceptable  team  size  for  a  SCAMPI  appraisal  is  four  people 
(including  the  team  leader).  The  maximum  recommended  team  size  is  nine. 

All  team  members  must  have  previously  completed  the  SEI-licensed 
Introduction  to  CMMI  course,  delivered  by  an  instructor  who  is  authorized  by 
the  SEI. 

Team  members’  training  in  the  appraisal  method  is  discussed  in  activity  1.3.3, 
Prepare  Team. 

With  regard  to  engineering  field  experience,  the  team  (as  a  group)  must  have 
an  average  of  at  least  6  years  of  experience,  and  the  team  total  must  be  at  least 
25  years  of  experience,  in  each  of  the  disciplines  to  be  covered  in  the 
appraisal. 

With  regard  to  management  experience,  the  team  (as  a  group)  must  have  a 
total  of  at  least  10  years  of  experience,  and  at  least  one  team  member  must 
have  at  least  6  years  of  experience  as  a  manager. 

The  team  should,  in  aggregate,  have  representative  experience  in  the  life 
cycles  in  use  within  the  appraised  organization.  For  any  given  life-cycle 
phase,  at  least  two  members  of  the  team  should  have  experience  as  a 
practitioner. 

Team  members  should  not  be  managers  of  one  of  the  selected  projects  or  be 
within  the  direct  supervisory  chain  of  any  of  the  anticipated  interviewees. 
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1 .3.2  Select  Team  Members  (continued) 


Optional  Although  not  required  in  the  Parameters  and  Limits  section  above,  the 

Practices  following  are  considered  recommended  best  practices  and  should  be 

employed  whenever  feasible: 

•  Each  member  should  have  good  written  and  oral  communication  skills, 
the  ability  to  facilitate  the  free  flow  of  communication,  and  the  ability  to 
perform  as  team  players  and  negotiate  consensus. 

•  At  least  half  of  the  team  members  should  have  participated  in  a  previous 
process  appraisal. 

•  Team  members  should  be  perceived  by  the  appraisal  sponsor  as  credible. 

Additional  appraisal  team  member  selection  considerations: 

•  Consider  the  personal  characteristics  of  individual  team  members  (e.g., 
communication  preferences,  personality  types)  and  how  these  may  affect 
the  dynamics  in  a  team  environment. 

•  Use  one  or  more  authorized  Lead  Appraisers  as  team  members. 


Implementation  Appraisal  team  members  are  selected  to  provide  a  diverse  set  of  qualified 
Guidance  professionals  with  the  appropriate  experience,  knowledge,  and  skills  to  make 

reasoned  judgments  regarding  implementation  of  the  reference  model. 

The  accuracy  and  credibility  of  the  appraisal  results  depends  greatly  on  the 
capability,  qualifications,  and  preparation  of  the  appraisal  team  members.  In 
addition  to  the  qualifications  described  above,  other  factors  that  may  affect 
the  performance  of  the  team  or  reliability  of  appraisal  results  should  be 
considered.  Appraisal  constraints,  such  as  security  classification,  may  result  in 
additional  criteria  for  team  member  selection. 

The  selected  appraisal  team  members  and  their  organizational  affiliation  and 
qualifications  (individually  and  in  aggregate)  are  documented  in  the  appraisal 
plan.  Appraisal  team  members  are  typically  selected  from  a  pool  of  qualified 
individuals  provided  by  the  appraisal  sponsor  or  his/her  designee.  The 
appraisal  team  leader  is  the  final  authority  on  acceptance  of  appraisal  team 
members,  and  is  responsible  for  ensuring  their  qualifications  and  suitability 
for  the  appraisal  purpose. 

Situations  where  a  conflict  of  interest  may  arise  should  be  avoided.  Team 
members  who  manage  people  or  processes  in  the  organization  may  struggle 
with  their  ability  to  be  objective.  Team  members  who  are  directly  impacted  by 
the  appraisal  outcome  may  be  distracted  by  the  potential  consequences  of  the 
decisions  they  contribute  to  on  the  appraisal  team. 


♦ 


CMU/SE1-2001-HB-001 


Page  1I-39 


1 .3.3  Prepare  T earn 


Activity 

Description 


Required 

Practices 


The  appraisal  team  leader  is  responsible  for  ensuring  that  appraisal  team 
members  are  sufficiently  prepared  for  performing  the  planned  appraisal 
activities.  This  includes  familiarity  with  the  reference  model,  SCAMPI,  the 
appraisal  plan,  organizational  data  and  characteristics,  and  the  tools  and 
techniques  to  be  used  during  the  appraisal.  Roles  and  responsibilities  are 
assigned  for  appraisal  tasks.  Team  building  exercises  are  used  to  practice 
facilitation  skills  and  reach  unity  in  understanding  the  team  objectives  and 
how  they  will  be  satisfied. 

All  team  members  are  expected  to  observe  strict  rules  for  confidentiality,  the 
protection  of  proprietary  or  sensitive  data,  and  the  non-attribution  of 
information  to  project  participants.  Non-disclosure  statements  are  often  used 
to  formalize  these  understandings. 


•  Ensure  that  appraisal  team  members  have  received  reference  model 
training. 

•  Provide  appraisal  method  training  to  appraisal  team  members  or  ensure 
that  they  have  already  received  it. 

•  Provide  for  team  building  and  establishing  team  norms. 

•  Provide  orientation  to  team  members  on  appraisal  objectives,  plans,  and 
their  assigned  roles  and  responsibilities. 


Continued  on  next  page 
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1 .3.3  Prepare  Team  (continued) 


Parameters 
and  Limits 


Model  training  must  be  provided  using  the  standard  Introduction  to  CMMI 
course,  delivered  by  an  instructor  who  is  authorized  by  the  CMMI  Steward. 

Method  training  may  be  delivered  in  one  of  two  ways: 

1.  method  training  specific  to  the  appraisal  at  hand 

2.  method  training  delivered  to  a  large  group  of  potential  future  team 
members  who  are  not  currently  engaged  in  an  appraisal 

Method  training  delivered  to  an  intact  team  must  be  at  least  two  days  in 
duration  and  must  emphasize  the  situations  likely  to  be  encountered  by  team 
members  during  the  appraisal.  This  training  will  not  necessarily  cover  all 
variants  in  the  application  of  SCAMPI. 

Method  training  delivered  to  groups  of  potential  future  team  members  must 
cover  the  complete  set  of  tailoring  options  and  allowable  variations  for  the 
method  to  prepare  them  for  a  range  of  situations  they  are  likely  to  encounter 
on  future  appraisals.  The  SEI  Appraiser  Program  specifies  additional 
requirements  about  delivering  training  to  people  who  are  not  already  members 
of  an  appraisal  team. 

Team  members  who  have  previously  been  trained  as  a  member  of  a  prior 
appraisal  team  are  not  automatically  qualified  to  participate  on  a  subsequent 
appraisal  without  first  attending  method  training.  In  such  cases,  the  appraisal 
team  leader  is  required  to  understand  the  nature  of  the  training  delivered 
previously  and  the  adequacy  of  that  training  for  the  appraisal  at  hand.  This 
requires  that  the  previous  appraisal  be  compared  with  the  planned  appraisal. 
For  example,  if  the  team  member  participated  in  an  appraisal  focused  only  on 
software  engineering,  using  the  continuous  representation,  and  the  planned 
appraisal  is  focused  on  SE/SW/IPPD  using  a  staged  representation,  there  may 
be  some  important  new  concepts  to  cover  with  that  team  member. 

There  must  be  at  least  one  event  where  the  team  gathers  as  a  group  for  the 
purpose  of  establishing  team  norms  and  operational  decisions  about  how  the 
team  will  work  for  the  appraisal  at  hand. 
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1.3.3  Prepare  Team  (continued) 


Optional 

Practices 


Implementation 

Guidance 


Implementation 

Guidance 

Training  in 
the  Reference 
Model 


Some  organizations  have  established  an  “organic”  capability  to  perform 
appraisals  with  very  limited  preparation  effort,  through  the  use  of  a  pool  of 
trained  appraisal  team  members.  Drawing  from  an  established  group  of 
experts,  who  are  accustomed  to  working  together,  clearly  provides  a  savings 
over  time  for  organizations  that  conduct  frequent  appraisals. 


The  team  training  event  is  a  good  place  to  review  the  appraisal  plan  with 
appraisal  team  members,  having  sent  it  to  them  in  advance  of  their  arrival. 
This  event  provides  the  orientation  for  the  entire  appraisal  that  all  appraisal 
team  members  need  to  execute  their  roles  appropriately.  This  also  is  in 
keeping  with  the  “Provide  appraisal  plan  to  relevant  stakeholders  for  review” 
required  practice  in  activity  1.2.6. 

Additionally,  the  team  training  event  is  a  primary  opportunity  to  conduct 
activity  1.5.1,  Perform  Readiness  Review.  The  assembled,  trained  appraisal 
team  can  then  appropriately  assess  the  organization’s  readiness  for  the 
appraisal  and  validate  the  reasonableness  of  appraisal  estimates. 


A  typical  model  training  course  is  delivered  in  two-and-a-half  to  three  days. 
Although  training  in  either  model  representation  (staged  or  continuous)  is 
allowable,  it  is  recommended  that  this  training  be  provided  for  the  model 
representation  to  be  used  during  the  appraisal.  The  successful  completion  of 
reference  model  training  should  precede  training  in  the  appraisal  method. 
There  is  no  “aging”  requirement  for  when  this  model  training  was  received, 
but  the  appraisal  team  leader  is  responsible  for  ensuring  that  each  team 
member  has  adequate  reference  model  understanding,  and  for  taking  remedial 
action  if  necessary.  Attendance  at  model  training  needs  to  be  recorded  by  the 
training  instructor  and  provided  to  the  CMMI  Steward,  in  accordance  with  the 
terms  of  the  instructor  authorization. 

For  appraisals  that  include  higher  levels  (i.e.,  maturity/capability  levels  4  and 
5)  team  members  may  benefit  from  receiving  additional  training  on  this 
subject  matter.  The  Intermediate  Concepts  of  CMMI  course,  a  course  on 
Statistical  Process  Control,  and/or  other  advance  topics  may  be  of  use  for  this 
added  level  of  preparation. 
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1 .3.3  Prepare  Team  (continued) 


Implementation 

Guidance 

Training  in 
the  Appraisal 
Method 


Implementation 

Guidance 

Famiiiarization 
with  the 
Appraisai  Plan 


A  typical  delivery  of  appraisal  team  training  might  take  two-and-a-half  to 
three  days.  More  or  less  time  may  be  necessary  depending  on  the  relative 
experience  of  the  appraisal  team  members. 

Exercises  in  appraisal  techniques  and  team  development  are  used  to  reinforce 
the  skills  that  will  be  important  during  conduct  of  the  appraisal.  It  is 
recommended  that  exercises  be  used  that  are  appropriate  for  the 
organizational  unit  being  appraised.  Where  sufficient  organizational  artifacts 
exist,  “live”  data  can  be  collected  and  used  in  training  exercises  where 
appropriate.  Just-in-time  training  can  also  be  used  to  re-emphasize  method 
concepts  at  appropriate  points  in  the  appraisal  process  during  which  the  skills 
will  be  utilized. 

Appraisal  team  training  materials  should  be  tailored  to  fit  team  needs  and 
objectives  of  the  specific  appraisal.  Tailoring  provides  opportunities  to 

•  provide  insight  into  the  context,  objectives,  and  plans  of  the  particular 
appraisal 

•  communicate  team  members’  assigned  roles  and  responsibilities 

•  identify  tailoring  of  SCAMPI  for  the  upcoming  appraisal 

•  acquaint  the  team  with  the  organizational  unit’s  characteristics  and 
documentation 

•  focus  on  skills  that  may  be  more  critical  to  the  upcoming  appraisal,  such 
as  the  ability  to  facilitate  interviews  or  the  ability  to  identify  alternative 
practices 

It  is  recommended  that  this  training  be  provided  within  60  days  of  the 
appraisal.  The  appraisal  team  leader  typically  provides  method  training,  but 
other  delivery  options  are  also  acceptable  (as  described  above).  Although 
alternative  training  options  can  provide  some  advantages  and  efficiencies  for 
method  training,  there  are  also  potential  consequences  that  might  be  felt  by 
the  appraisal  team  leader  on  a  given  appraisal,  such  as  poor  training  quality  or 
readiness  of  team  members.  Regardless  of  how  method  training  is  delivered  to 
the  team  members,  opportunities  for  team  building  should  be  provided  to 
coalesce  the  team  and  bring  the  team  up  to  speed  on  the  specifics  of  the 
appraisal  being  planned. 


Method  training  and  team  building  provide  good  opportunities  to  establish 
team  familiarity  with  the  appraisal  plan.  This  includes  such  items  as  appraisal 
objectives,  organizational  scope,  reference  model  scope,  and  the  schedule, 
resources,  and  constraints  for  conducting  the  appraisal.  Team  member  input 
can  be  obtained  to  refine  or  complete  the  contents  of  the  appraisal  plan. 
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1 .3.3  Prepare  Team  (continued) 


Implementation 

Guidance 

Analysis  of 

Objective 

Evidence 


Analysis  of  the  objective  evidence  provided  by  the  appraised  organization, 
such  as  questionnaire  responses  or  worksheets  summarizing  objective 
evidence,  can  be  accomplished  following  or  as  an  integrated  part  of  appraisal 
team  preparation  and  training. 

Team  members  should  become  familiar  with  the  instruments  (e.g., 
questionnaires,  PII  database)  to  be  used  as  data  collection  sources  during  the 
appraisal.  Demonstrations  or  exercises  using  the  data  collection  tools  and 
methods  planned  for  the  appraisal  should  be  used  to  provide  appraisal  team 
members  with  an  opportunity  to  practice  techniques  for  data  recording, 
verification,  and  analysis.  This  may  include  mechanisms  such  as  wall  charts, 
spreadsheets,  or  data  reduction  tools.  The  more  familiarity  and  comfort  that 
can  be  obtained  with  these  tools  in  advance,  the  greater  the  savings  in  team 
efficiency  during  the  appraisal  on-site  phases. 


Continued  on  next  page 


Page  11-44 


CMU/SEI-2001-HB-001 


1 .3.3  Prepare  Team  (continued) 


Implementation 

Guidance 

Roles  and 
Responsibilities 


The  appraisal  team  leader  should  assign  and  explain  team  member  roles  and 
responsibilities  to  be  performed  during  the  appraisal.  Typical  roles  to  be 
assigned  include: 

Organizational  Unit  Coordinator.  The  Organizational  Unit  Coordinator 
handles  on-site  logistics  and  provides  technical,  administrative,  and  logistical 
support  to  the  appraisal  team  leader.  This  usually  includes  activities  such  as 
coordinating  schedules,  notifying  participants,  arranging  adequate  facilities 
and  resources,  obtaining  requested  documentation,  and  arranging  catering.  He 
or  she  may  also  coordinate  or  provide  clerical  support  to  the  team.  This  role  is 
often  assigned  to  one  or  more  members  of  the  organizational  unit.  The 
Organizational  Unit  Coordinator  may  be  one  of  the  appraisal  team  members, 
or  this  role  may  be  assigned  to  other  site  personnel. 

Librarian:  The  librarian  manages  the  inventory  of  appraisal  documents, 
coordinates  requests  for  additional  documentation  evidence,  and  returns 
documents  at  the  end  of  the  appraisal.  This  role  can  be  fulfilled  by  an 
appraisal  team  member  or  by  a  member  of  the  support  staff. 

Process  Area  Mini-Teams:  Mini-teams  take  the  lead  for  data  collection  in 
assigned  PAs.  They  ensure  that  information  collected  during  a  data  gathering 
session  covers  their  PAs,  request  additional  information  needed  relative  to 
their  PAs,  and  record  the  work  performed  by  individual  appraisal  team 
members  pertaining  to  their  PAs. 

Mini-teams  typically  consist  of  two  or  three  members.  Mini-team  assignments 
can  be  made  based  on  several  factors,  including 

•  related  PAs  (e.g.,  PA  categories) 

•  composition  mix  of  mini-team  members  (e.g.,  discipline  experience, 
appraisal  experience) 

Facilitator:  The  facilitator  conducts  interviews,  asking  questions  of  interview 
participants. 

Timekeeper.  The  timekeeper  is  responsible  for  tracking  time  and  schedule 
constraints  during  interviews  and  other  activities. 

Observer:  Due  to  the  confidentiality  required  during  an  appraisal  and  the 
cohesiveness  needed  to  participate  in  appraisal  activities,  observers  are  not 
permitted  to  participate  in  the  appraisal  processes.  The  only  exception  is  an 
observer  who  is  authorized  by  the  CMMI  Steward  to  observe  a  candidate 
Lead  Appraiser’s  performance  as  appraisal  team  leader  or  to  perform  an  audit 
as  part  of  the  quality  audit  function  of  the  Steward. 
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1 .4  Obtain  and  Analyze  Initial  Objective  Evidence 


Purpose  Obtain  information  that  facilitates  site-specific  preparation  and  an 

understanding  of  the  implementation  of  model  practices  across  the 
organizational  unit.  Identify  potential  issues,  gaps,  or  risks  to  aid  in  refining 
the  plan.  Strengthen  understanding  of  the  organization’s  operations  and 
processes. 

Entry  Criteria  •  Appraisal  input  received 

•  Sponsor  authorization  to  proceed 

•  Availability  of  practice  implementation  data  for  organizational  unit 

Inputs  •  Practice  implementation  data  for  organizational  unit 

•  Identified  participants 

Activities  1.4.1  Prepare  Participants 

1.4.2  Administer  Instruments 

1.4.3  Obtain  Initial  Objective  Evidence 

1.4.4  Inventory  Objective  Evidence 

Outputs  •  Completed  instruments 

•  Data  analyses  results  (data  summaries,  questionnaire  results,  etc.) 

•  Identification  of  additional  information  needed 

•  Prepared  participants 

•  Initial  set  of  objective  evidence 

Outcome  •  Initial  objective  evidence  has  been  collected,  organized,  and  recorded. 

•  Potentially  important  areas  of  needed  information  have  been  noted. 

•  The  team  has  a  deeper  understanding  of  the  organizational  unit’s 
operations  and  processes. 

•  The  team  is  ready  to  make  detailed  plans  for  data  collection. 

Exit  Criteria  •  All  objective  evidence  captured  during  this  activity  has  been  recorded  for 
later  use. 

•  High-priority  areas  for  additional  data  collection  have  been  identified. 

•  The  level  of  sufficiency  of  the  objective  evidence  to  support  the  appraisal 
is  determined. 

Continued  on  next  page 
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1.4  Obtain  and  Analyze  Initial  Objective  Evidence  (continued) 


Key  Points 

Gather  high-leverage  objective  evidence.  The  amount  of  initial  objective 
evidence  provided  by  the  organization  will  determine  the  proportion  of 
evidence  that  must  be  discovered  (versus  verified)  during  the  appraisal. 
Maximizing  time  spent  in  verification,  versus  discovery,  is  a  key  performance 
objective  for  the  appraisal  process. 

Tools  and 
Techniques 

•  Automated  support  for  questionnaires,  including  data  reduction  tools,  may 
be  available  to  make  the  data  analysis  activity  more  efficient. 

•  Breaking  into  mini-teams  to  review  data  related  to  specific  PAs  is  a  way 
to  ensure  completeness  in  the  data. 

Metrics 

•  The  number  of  practices  for  which  complete  objective  evidence  is 
available 

•  The  number  of  questionnaire  respondents  reported  in  the  Appraisal 
Record 

•  The  calendar  time  and  effort  expended  for  this  activity  compared  to  the 
planned  values 

Verification  and 
Validation 

•  Where  the  team  includes  members  of  the  appraised  organization,  these 
members  should  be  used  to  help  understand  the  initial  objective  evidence 
provided  to  prevent  misinterpretation  of  terms  or  special  conditions. 

•  Inconsistencies  and  contradictions  among  the  items  provided  in  initial 
objective  evidence  should  be  identified  and  recorded  for  resolution. 

Records 

•  Records  of  this  process  include  completed  and/or  summarized 
questionnaires,  profiles,  and  surveys. 

•  Lists  of  information  needed  should  be  maintained  and  used  as  an  input  to 
the  later  data  collection  activities. 

•  Calendar  time  and  effort  expended  in  this  activity  should  be  recorded  and 
compared  to  the  plan.  These  data  will  be  part  of  the  Appraisal  Record. 

Tailoring 

A  variety  of  methods  can  be  used  to  collect  initial  data,  including 

•  a  site  information  package  prepared  by  representatives  of  the  organization 

•  a  presentation  on  the  process  inqwovement  program  and  its  accomplishments 

•  specialized  or  general  questionnaires  focused  on  practice  implementation 

The  use  of  additional  instruments  is  dependent  on  the  results  of  the  analysis  of 
available  data  and  the  results  of  process  1.5,  Prepare  for  Collection  of 
Objective  Evidence. 
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1.4  Obtain  and  Analyze  Initial  Objective  Evidence  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


This  process  plays  a  critical  role  in  the  planning  and  preparation  processes. 
The  information  generated  in  this  process  provides  the  most  important 
opportunity  to  reset  expectations  and  plans  with  the  appraisal  sponsor,  if 
initial  assumptions  about  the  availability  of  objective  evidence  turn  out  to  be 
in  error.  It  will  also  provide  the  basis  of  data  collection  planning. 


The  appraisal  team  leader  works  with  representatives  of  the  organization  to 
obtain  an  initial  data  set  that  represents  an  inventory  of  the  objective  evidence 
pertaining  to  the  implementation  of  each  instantiation  of  each  practice  within 
the  appraisal  scope.  This  initial  data  set  is  first  reviewed  by  the  appraisal  team 
leader  for  a  high-level  assessment  of  adequacy  and  completeness.  The 
appraisal  team  leader  or  appraisal  team  then  performs  a  more  detailed  analysis 
to  use  as  input  for  planning  the  data  collection  and  verification  activities  that 
will  occur  when  they  arrive  on  site.  Finally,  a  record  is  created  that  reflects  a 
detailed  accounting  of  any  missing  objective  evidence.  This  record  is  used  as 
primary  input  for  the  generation  of  the  data  collection  plan. 
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1.4.1  Prepare  Participants 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


Members  of  the  organization  who  participate  in  the  appraisal  must  be 
informed  of  their  role,  and  the  expectations  the  sponsor  and  appraisal  team 
have.  This  is  typically  accomplished  through  a  briefing  where  the  appraisal 
team  leader  provides  an  overview  of  the  appraisal  process,  purpose,  and 
objectives.  Specific  information  about  the  scheduled  events  and  the  locations 
where  they  occur  is  also  communicated  during  this  presentation,  as  well  as 
through  ongoing  contact  between  the  Organizational  Unit  Coordinator  and  the 
members  of  the  organization. 


•  Brief  appraisal  participants  on  the  appraisal  process. 

•  Provide  orientation  to  appraisal  participants  on  their  roles  in  the  appraisal. 


The  orientation  provided  to  appraisal  participants  must  occur  some  time  prior 
to  their  participation  to  allow  participants  to  confirm  their  availability  and  to 
prepare  for  their  participation. 

The  preparation  of  appraisal  participants  may  be  accomplished  via 
video/teleconference  if  desired. 
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1.4.1  Prepare  Participants  (continued) 


Optional  Provide  orientation  on  the  documentation  of  PIIs  and  any  specific  instruments 

Practices  used,  so  the  appropriate  people  in  the  organization  can  document  the  initial 

objective  evidence  to  be  used  in  the  appraisal. 


Implementation  Depending  on  the  appraisal  usage  mode  (e.g.,  supplier  selection  versus 
Guidance  internal  process  improvement),  various  types  of  communications  may  be 

used.  In  the  internal  process  improvement  usage  mode,  the  importance  of 
management  sponsorship  within  the  organization  will  likely  lead  the  appraisal 
team  leader  to  work  with  senior  management  to  help  demonstrate 
commitment  to  the  appraisal  process  as  well  as  the  process  improvement  work 
that  will  follow.  In  the  supplier  selection  usage  mode,  the  possibility  of  the 
same  team  visiting  multiple  organizations  adds  coordination  tasks  and 
communication  channels  as  well. 

Preparation  of  appraisal  participants  should  also  include  informing  them  of 
the  need  to  provide  accurate  and  complete  information  on  instruments.  In 
addition  to  assisting  with  appraisal  accuracy,  this  can  help  to  ensure  sufficient 
coverage  of  reference  model  practices  and  reduce  the  amount  of  time 
necessary  for  follow-up  interviews.  The  investment  in  initial  population  of 
complete  instruments,  such  as  PIIs,  questionnaires,  or  mapping  tables,  can  be 
recovered  by  reduced  effort  in  the  reuse  of  assets  for  subsequent  appraisals. 
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1.4.2  Administer  Instruments 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


This  activity  involves  the  administration  of  instruments  for  the  appraisal  that 
are  additional  to  the  input  data  (such  as  process  implementation  indicators 
provided  by  the  organization  as  input  to  the  appraisal).  It  includes  the  use  of 
structured  techniques  and  work  aids  (e.g.,  surveys,  questionnaires,  or  an 
objective  evidence  database)  to  assist  the  organizational  unit  in  characterizing 
their  process  and  supporting  objective  evidence  in  terms  of  model  practices. 

A  practice-based  questionnaire  is  also  a  commonly  used  instrument  during 
appraisals.  Such  questionnaires  typically  have  a  series  of  focused  questions, 
each  one  providing  an  opportunity  for  the  respondent  to  answer  a  closed- 
ended  question  about  a  practice.  In  addition,  the  respondent  is  given  an 
opportunity  to  write  a  clarifying  comment  that  serves  to  elaborate  on  the 
closed -ended  response. 


Administer  appraisal  instruments  for  the  entry  of  data  by  appraisal 
participants. 


The  application  of  this  activity  to  generate  instrument  data  to  support  the  data 
collection  plan  is  limited  to  the  instruments  identified  in  the  Data  Collection 
Plan.  Instruments  are  typically  administered  by  representatives  of  the 
appraisal  team.  The  appraisal  team  leader  is  responsible  for  negotiating 
additional  time  and  resources  if  the  data  provided  using  instruments  is 
incomplete.  It  is  also  the  responsibility  of  the  appraisal  team  leader  to  avoid 
requesting  duplicate  data  entry  on  multiple  instruments.  No  organization 
should  be  asked  to  provide  the  same  information  in  two  (or  more)  formats. 

Whatever  vehicle  is  used,  the  resultant  data  must  provide  information  about 
the  extent  of  the  implementation  of  model  practices  in  the  organizational  unit 
and  the  sampled  projects. 


Continued  on  next  page 
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1 .4.2  Administer  Instruments  (continued) 


Optional  Establish  an  organizational  asset  (or  rely  on  an  existing  one)  that  documents 

Practices  and  maintains  the  traceability  of  implemented  practices  to  model  practices. 

Conduct  a  workshop  to  document  the  PIIs  for  the  organization. 


Implementation  The  use  of  instruments  to  gather  written  information  from  members  of  the 
Guidance  organization  provides  a  relatively  low-cost  data  collection  technique  when 

done  well.  Data  of  this  type  tend  to  be  most  useful  when  provided  early  in  the 
appraisal  conduct,  and  can  lead  to  valuable  insights  about  where  data  may  be 
sought  during  subsequent  data  collection  events.  Since  there  is  limited 
opportunity  for  elaboration  and  “branching”  to  related  topics,  responses  to 
instruments  can  sometimes  raise  more  questions  than  they  answer  for  the 
appraisal  team  member  trying  to  interpret  the  responses.  Furthermore, 
instruments  that  contain  excessive  jargon  or  complicated  terminology  may 
hinder  data  collection  rather  than  help.  Confused  respondents  will  do  their 
best  to  answer  the  question  they  don’t  quite  understand,  and  the  response  is 
interpreted  based  on  the  question  that  was  intended.  Having  a  knowledgeable 
person  present  during  the  administration  of  an  instrument  can  help  mitigate 
the  risk  of  miscommunication. 

One  of  the  attractive  features  of  instruments  for  the  purpose  of  data  collection 
is  that  they  can  be  used  to  establish  a  “scoring  scheme”  that  reduces  the 
burden  of  interpretation  for  the  recipient  of  the  data.  Such  schemes  do  not 
exist  for  SCAMPI,  and  the  use  of  a  shortcut  of  this  type  is  a  violation  of  the 
principle  that  focuses  rating  judgments  on  the  goals  of  the  PAs  in  CMMI 
models.  The  practices  found  in  CMMI  models  are  Expected  Components, 
while  the  goals  in  the  models  are  Required  Components.  While  the 
satisfaction  of  a  PA  goal  is  predicated  on  the  implementation  of  practices 
found  in  the  model  (or  acceptable  alternatives),  there  is  no  strict  aggregation 
scheme  that  allows  one  to  infer  goal  satisfaction  based  on  practice 
implementation  alone.  Rating  judgments  are  based  on  multiple  sources  of 
objective  evidence  and  the  reasoned  consideration  of  strengths  and 
weaknesses,  in  aggregate. 

Whenever  possible,  documents  mentioned  in  the  responses  to  questionnaires 
or  other  instruments  should  be  requested  for  team  review  early  in  the  process, 
so  that  any  misleading  references  will  not  cause  undue  confusion  later. 


♦ 


CMU/SEI-2001-HB-001 


Page  11-53 


1.4.3  Obtain  Initial  Objective  Evidence 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


Optional 

Practices 


The  appraisal  team  leader  will  request  that  the  organization  provides  detailed 
data  on  the  implementation  of  practices  in  the  organization.  The  appraisal 
team  leader  is  free  to  specify  the  format  to  be  used  and  the  level  of  detail  to  be 
provided,  knowing  that  anything  that  is  not  provided  in  advance  must  be 
collected  later  in  the  appraisal  process.  There  are  no  minimum  requirements 
set  by  the  method  with  respect  to  completeness  or  detail  in  this  initial  data  set. 
However,  the  effort  required  to  conduct  a  SCAMPI  appraisal  is  a  direct 
function  of  the  amount  of  data  available  to  the  team  at  the  beginning  of  the 
process.  Before  the  appraisal  outputs  can  be  created,  the  team  will  need  to 
verify  objective  evidence  for  each  instantiation  of  each  practice  within  the 
scope  of  the  appraisal.  For  detailed  requirements  on  the  sufficiency  of  data, 
refer  to  process  2.2,  Verify  and  Validate  Objective  Evidence. 

The  use  of  a  completely  populated  PII  database  is  desirable  but  not  essential 
at  this  stage  in  the  appraisal  process.  The  appraisal  team  leader  must  provide 
an  opportunity  for  the  organization  to  provide  it,  but  will  not  require  it  unless 
the  sponsor  has  agreed  that  this  will  be  a  verification-oriented  appraisal  (as 
opposed  to  a  discovery-oriented  appraisal). 

A  “mapping”  of  implemented  practices  and  model  practices  is  required,  and 
may  be  generated  using  questionnaires  (see  activity  1.4.2). 


Obtain  documentation  reflecting  the  implementation  of  model  practices 
within  the  organizational  unit  and  sampled  projects. 


At  a  minimum,  the  organization  must  provide  a  list  of  documents  that  are 
relevant  to  understanding  the  processes  in  use  in  the  organizational  unit  and 
the  sampled  projects.  This  list  must  be  mapped  to  the  model  practices  that  are 
in  the  scope  of  the  appraisal. 


A  list  of  terms  and  important  jargon  used  in  the  organizational  unit  may  be 
provided  to  the  team,  to  aid  in  communicating  with  the  members  of  the 
organization. 

A  complete  objective  evidence  database,  which  documents  the 
implementation  of  every  model  practice  (within  the  scope  of  the  appraisal)  in 
the  organizational  unit  and  the  sampled  projects,  may  be  provided  to  the  team 
in  advance. 

The  use  of  database  tools  specifically  built  to  support  a  process  appraisal  is 
highly  recommended. 


Continued  on  next  page 
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1.4.3  Obtain  Initial  Objective  Evidence  (continued) 


Implementation 

Guidance 


Whether  collected  through  instruments,  the  review  of  documents,  attending 
presentations,  or  interviews,  the  data  used  for  an  appraisal  is  related  to  the 
practices  of  the  reference  model.  For  every  practice  within  the  model  scope  of 
the  appraisal,  and  for  every  instance  of  each  practice,  objective  evidence  is 
used  as  the  basis  for  appraisal  team  determinations  of  the  extent  to  which  the 
practice  is  implemented.  Indicators  that  substantiate  practice  implementation 
include 

•  direct  artifacts,  which  represent  the  primary  tangible  output  of  a  practice. 
These  are  typically  listed  in  CMMI  models  as  typical  work  products.  One 
or  more  direct  artifacts  may  be  necessary  to  verify  the  implementation  of 
associated  model  practices. 

•  indirect  artifacts,  which  represent  artifacts  that  are  a  consequence  of 
performing  the  practice,  but  not  necessarily  the  purpose  for  which  it  is 
performed.  These  are  typically  things  like  meeting  minutes,  review 
results,  or  written  communications  of  status. 

•  affirmations,  which  are  oral  or  written  statements  confirming  the 
implementation  of  the  practice.  These  are  typically  validated  using 
interviews,  questionnaires,  or  other  means. 

Prior  to  the  data  collection  activities  carrieji  out  by  the  appraisal  team,  an 
initial  data  set  is  usually  created  by  the  appraised  organization.  This  data  set 
contains  descriptions  of  the  objective  evidence  available  for  the  team  to 
examine,  complete  with  references  to  documentation  and  identification  of  the 
personnel  who  can  provide  relevant  affirmations.  This  instrument  provides  the 
baseline  of  objective  evidence  for  the  appraisal.  Most  organizations 
experienced  in  process  improvement  will  already  have  this  type  of  data  on 
hand,  as  they  will  have  used  it  to  track  their  progress. 

Artifacts  may  be  obtained  as  hard  copies,  soft  copies,  or  hyperlinks  to  where 
these  documents  reside  in  a  Web-based  environment.  If  hyperlinks  are  used, 
the  accessibility  of  artifacts  via  these  links  should  be  verified  in  the  appraisal 
environment.  For  example,  appraisal  team  access  could  be  inhibited  by 
invalid  references  or  firewalls. 

The  initial  data  set  forms  the  basis  for  planning  the  data  collection  activities, 
including  interviews  and  presentations  on  site.  Any  objective  evidence  that  is 
not  identified  in  advance  of  the  team’s  arrival  will  need  to  be  sought  by  the 
team  members  once  they  arrive  on  site.  This  process  of  “discovering” 
whether,  and  how,  the  organization  has  addressed  a  given  practice  in  the 
model  can  be  quite  time  consuming,  and  it  is  often  difficult  to  predict  how 
long  it  will  take. 
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1.4.4  Inventory  Objective  Evidence 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


The  analysis  of  the  initial  data  set  provides  critical  new  information  for  the 
overall  planning  of  the  appraisal  and  forms  the  basis  for  the  detailed  data 
collection  plan  that  must  be  developed  before  the  on-site  data  collection 
begins.  The  analysis  of  initial  objective  evidence  at  this  stage  is  focused 
primarily  on  the  adequacy  and  completeness  of  information  and  the 
iniplications  for  future  data  collection.  The  results  of  this  analysis  will  be  the 
primary  basis  for  determining  the  extent  to  which  the  appraisal  will  be  one  of 
verification  or  discovery. 


•  Examine  the  initial  set  of  objective  evidence  provided  by  the 
organizational  unit. 

•  Detemune  the  extent  to  which  additional  information  is  needed  for 
adequate  coverage  of  model  practices. 


Information  provided  by  the  organizational  unit  must  be  detailed  enough  to 
understand  the  extent  to  which  each  type  of  objective  evidence  (direct 
artifacts,  indirect  artifacts,  and  affirmations)  is  available  for  each  process 
instantiation,  for  each  model  practice  within  the  scope  of  the  appraisal.  This 
initial  review  of  objective  evidence  identifies  model  practices  for  which  the 
team  has 

•  strong  objective  evidence 

•  no  objective  evidence 

•  conflicting  objective  evidence 

•  anomalous  objective  evidence 

•  insufficient  objective  evidence 

Key  documents  are  identified  that  can  be  used  to  gain  insight  regarding  a 
number  of  model  practices.  These  are  potential  high-leverage  documents  that 
may  be  good  candidates  for  pre-on-site  review  by  team  members. 


Review  the  initial  objective  evidence  with  members  of  the  engineering 
process  group. 


Continued  on  next  page 
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1.4.4  Inventory  Objective  Evidence  (continued) 


Implementdtion  Members  of  the  team  may  choose  to  summarize  the  extent  of  practice 
Guidance  implementation  at  the  discretion  of  the  appraisal  team  leader.  However,  the 

objective  of  this  activity  is  to  determine  how  much  additional  data  team 
members  will  need  to  complete  their  work.  It  is  recommended  that  the 
appraisal  team  leader  establish  an  expectation  with  the  sponsor  that  the  results 
of  this  analysis  will  form  the  basis  for  a  revised  schedule  estimate.  If  the 
initial  objective  evidence  is  lacking  in  completeness  and  detail,  the  team  will 
be  forced  to  seek  more  information  during  the  on-site  data  collection,  unless 
corrective  actions  are  taken  before  that  time. 

It  is  important  to  keep  all  stakeholders  focused  on  the  fact  that  SCAMPI  is 
intended  as  a  benchmarking  appraisal.  This  method  is  not  well  suited  for 
organizations  that  have  very  limited  understanding  of  CMMI.  Such 
organizations  may  not  yet  have  a  clear  idea  of  how  the  practices  described  in 
CMMI  models  ought  to  be  implemented  to  meet  their  specific  business  needs. 
Deciding  on  a  reasonable  implementation  of  the  practices,  and  working  to 
ensure  that  they  are  enacted  on  projects  throughout  the  organization,  are 
activities  that  precede  a  benchmarking  appraisal.  A  different  type  of  appraisal 
(Class  B  or  C)  is  probably  going  to  be  more  valuable  if  the  objective  of  the 
sponsor  is  to  begin  the  process  of  understanding  what  CMMI  could  mean  for 
the  organization.  It  is  not  reasonable  to  schedule  a  two-week  appraisal  and 
expect  to  collect  all  of  the  data  required  for  benchmarking  during  the  on-site 
data  collection. 

The  appraisal  team  leader  often  reviews  the  initial  data  set  provided  by  the 
organization  prior  to  assembling  the  team  for  its  first  meeting,  to  identify 
areas  where  additional  data  will  be  needed  and  to  assess  the  feasibility  of  the 
planned  appraisal  schedule.  This  readiness  review  should  be  conducted  prior 
to  finalizing  the  appraisal  schedule,  and  may  comprise  a  “Go/No  Go”  decision 
for  the  appraisal  in  some  situations.  The  appraisal  team  will  then  review  the 
initial  objective  evidence  in  more  detail  (typically  toward  the  end  of  the  team¬ 
training  event)  to  begin  formulating  plans  for  how  missing  evidence  will  be 
collected,  and  for  the  verification  of  the  entire  data  set.  This  preliminary 
readiness  review  is  the  basis  for  the  data  collection  plan,  which  is  described  in 
the  next  process.  Prepare  for  Collection  of  Objective  Evidence. 
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1.4.4  Inventory  Objective  Evidence  (continued) 


The  appraisal  team  leader  generates  a  list  of  additional  information  needed. 
The  results  of  the  analysis  of  initial  objective  evidence  are  documented  as  an 
input  to  the  data  collection  plan.  The  use  of  an  integrated  appraisal  tool  to 
annotate  the  set  of  initial  objective  evidence  will  permit  the  automated 
tracking  of  information  needs,  and  will  aid  in  the  compilation  of  a  detailed 
data  collection  plan.  Where  the  completeness  of  initial  objective  evidence  is 
insufficient  to  conduct  the  appraisal  under  the  original  schedule,  the  results  of 
this  activity  form  an  important  basis  for  renegotiating  the  appraisal  schedule 
in  some  cases. 

The  adequacy  of  objective  evidence  relative  to  model  practices  is  typically 
determined  using  a  software  tool  of  some  sort,  either  one  built  for  use  on 
appraisals,  or  a  spreadsheet  template.  However,  paper  forms  and  wall  charts 
may  be  used  if  preferred. 

♦ 


implementation 

Guidance 
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1 .5  Prepare  for  Collection  of  Objective  Evidence 


Purpose 

Entry  Criteria 

inputs 

Activities 

Outputs 

Outcome 

Exit  Criteria 


Plan  and  document  specific  data  collection  strategies,  including 

•  sources  of  data 

•  tools  and  techniques  to  be  used 

•  contingencies  to  manage  risk  of  insufficient  data 


•  Sponsor  commitment  to  proceed  with  the  appraisal  has  been  documented. 

•  Appraisal  objectives  and  constraints  have  been  documented. 

•  Initial  data  have  been  received  and  analysis  has  been  completed. 


•  Appraisal  plan 

•  PIIs  for  the  organizational  unit 

•  Initial  objective  evidence  review 

•  Data  collection  status 


1.5. 1  Perform  Readiness  Review 

1.5.2  Prepare  Data  Collection  Plan 

1.5.3  Replan  Data  Collection 


•  Confirmation  that  objective  evidence  collected  is  sufficient  to  proceed 

•  Initial  data  collection  plan 

•  Updates  to  the  plan  as  required 


Finalized  data  collection  plan.  Team  members  are  aware  of  data  needs  and  the 
status  of  initial  data  available  to  them. 


All  preparations  for  data  collection  by  the  team  have  been  made  and  the  data 
collection  plan  has  been  documented. 


Continued  on  next  page 
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1 .5  Prepare  for  Collection  of  Objective  Evidence  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification  and 
Validation 


Records 


Tailoring 


The  data  collected  is  the  most  important  input  the  team  receives.  Careful 
planning,  disciplined  tracking  against  the  plan,  and  effective  corrective 
actions  are  cornerstones  to  success  in  this  process. 


The  use  of  a  spreadsheet  to  record  and  track  the  data  collection  plan  is  a 
common  technique.  A  matrix  showing  the  practices  of  the  model,  or  questions 
to  be  asked,  arrayed  on  the  vertical  axis  and  the  sources  of  information 
arrayed  on  the  horizontal  axis  provides  a  simple  planning  and  tracking  tool.  A 
number  of  vendor-provided  data  management  tools  are  available  as  well. 


•  Estimated  and  tracked  calendar  time  and  effort  for  this  activity 

•  Planned  and  actual  number  of  data  sources  per  practice 

•  Planned  and  tracked  number  of  scripted  questions  per  interview 

•  Planned  and  tracked  number  of  scripted  questions  per  PA 

•  Percentage  of  planned  coverage  achieved,  per  data  collection  event  or  PA 


The  data  collection  plan  should  be  summarized  and  reviewed  with  the  team  to 
ensure  that  appraisal  requirements  will  be  successfully  implemented  if  the 
plan  is  carried  forward.  Experienced  Lead  Appraisers  will  use  historical  data 
to  assess  the  feasibility  of  (and  risks  associated  with)  the  data  collection  plan. 


Planned  and  actual  coverage  of  practices  and  PAs  across  the  set  of  data 
collection  activities  should  be  recorded.  These  data  support  future  estimates 
and  corrective  actions  during  the  data  collection  activities. 


Replanning  is  performed  only  when  the  status  of  the  appraisal  conduct 
indicates  the  need  to  do  so. 

Additional  planning  and  coordination  steps  may  be  necessary  in  usage  modes 
where  data  collection  activities  will  occur  at  geographically  distributed  sites 
or  across  organizational  units  from  different  corporations  (such  as  in  a 
Supplier  Selection  usage  mode). 

SCAMPI  allows  great  flexibility  in  formulating  strategies  to  accomplish  the 
necessary  data  collection.  The  relative  emphasis  of  different  data  sources,  as 
well  as  data  types,  can  be  tuned  to  support  appraisal  objectives  relating  to 
buy-in  as  well  as  coverage  and  rigor  for  important  areas. 
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1 .5  Prepare  for  Collection  of  Objective  Evidence  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  data  collection  plan  is  an  essential  element  of  the  appraisal  plan.  The 
activities  described  here  rely  on  the  results  of  analyzing  the  initial  objective 
evidence  to  derive  a  plan  and  set  of  strategies  for  accomplishing  the  data 
collection  needed  to  meet  the  objectives  of  the  appraisal.  The  data  collection 
plan  developed  through  these  activities  is  reviewed  and  revised  on  a  continual 
basis  throughout  the  appraisal.  Dynamically  managing  the  inventory  of  data 
on  hand,  the  list  of  data  needed,  and  the  available  data  collection  opportunities 
are  processes  critical  to  the  success  of  the  appraisal. 


The  activities  in  this  process  serve  to  (a)  establish  the  initial  planning  baseline 
for  the  acquisition  of  objective  evidence  and  (b)  update  the  plan  to  account  for 
information  acquired  and  unexpected  developments.  Since  SCAMPI  is  a  data- 
intensive  method,  the  conduct  of  these  activities  in  accordance  with  the 
descriptions  provided  is  essential  to  the  successful  use  of  the  appraisal 
method. 
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1.5.1  Perform  Readiness  Review 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optionai 

Practices 


The  available  objective  evidence  is  reviewed  to  determine  the  extent  to  which 
the  requested  objective  evidence  has  been  gathered  and  whether  the  evidence 
is  sufficient  to  proceed  or  replanning  is  required. 


•  Determine  whether  the  objective  evidence  for  each  process  instance  is 
adequate  to  proceed  with  the  appraisal  as  planned. 

•  Review  the  feasibility  of  the  appraisal  plan  in  light  of  the  inventory  of 
objective  evidence  available. 


At  least  one  readiness  review  must  be  conducted  prior  to  assembling  the  team 
on  site  for  data  collection. 

Objective  evidence  for  all  PAs  within  the  scope  of  the  appraisal  must  be 
reviewed. 

Objective  evidence  for  all  projects  sampled  to  represent  the  organizational 
unit  must  be  reviewed. 


Integrating  a  readiness  review  with  the  team  training  event  will  allow  the 
appraisal  team  leader  to  involve  the  team  in  gaining  an  understanding  of  the 
data  available  to  support  the  appraisal. 
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1.5.1  Perform  Readiness  Review  (continued) 


Implementation 

Guidance 


A  summary  of  the  inventory  of  objective  evidence  and  readiness  to  proceed 
should  be  reviewed  with  the  sponsor  or  his/her  designee.  If  insufficient 
objective  evidence  is  available,  the  appraisal  team  leader  may  need  to  initiate 
replanning  in  light  of  newly  discovered  constraints  (i.e.,  insufficient  data  to 
support  the  appraisal  as  planned).  Refer  to  activity  1.1.2,  Determine  Appraisal 
Constraints.  The  criteria  for  adequacy  will  depend  on  where  the  readiness 
review  occurs  in  the  schedule,  and  the  degree  of  verification  versus  discovery 
that  is  being  sought  for  the  on-site  phases  of  the  appraisal. 

More  than  one  readiness  review  is  likely  to  be  needed.  The  first  one  should  be 
performed  early  in  the  planning  phase,  and  the  second  should  be  performed 
once  the  objective  evidence  has  been  gathered  and  the  appraisal  is  ready  to 
start.  This  review  may  be  conducted  in  conjunction  with  the  team-training 
event. 

Thresholds  for  the  sufficiency  of  data  should  be  established  as  targets  to  be 
met  at  the  readiness  review.  For  example,  an  80%  threshold  may  be  used  to 
initiate  replanning  at  the  final  readiness  review.  That  is,  the  appraisal  team 
leader  establishes  an  expectation  with  the  sponsor  that,  if  more  than  20%  of 
the  objective  evidence  is  missing  at  the  time  of  team  training,  the  appraisal 
will  need  to  be  replanned.  However,  the  primary  objective  is  reducing  the  risk 
that  there  will  be  insufficient  objective  evidence  to  make  the  determinations 
required  by  the  appraisal  plan  in  the  time  allotted. 

The  readiness  review  is  a  key  event  whose  impact  should  not  be 
underestimated.  Failure  to  adequately  review  the  objective  evidence  available 
and  determine  the  impact  on  the  appraisal  plan  can  have  grave  consequences 
for  the  appraisal  team  during  the  on-site  period.  This  may  include  long  hours, 
exhaustion,  extensive  ad  hoc  data  collection  (discovery),  or  the  inability  to 
achieve  appraisal  objectives  within  defined  estimates  and  constraints. 


♦ 


CMU/SEI-2001-HB-001 


Page  11-63 


1.5.2  Prepare  Data  Collection  Plan 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  data  collection  activities  are  tailored  to  meet  the  needs  for  objective 
evidence  so  that  the  extent  of  practice  implementation  can  be  determined. 

For  practices  that  have  objective  evidence,  a  strategy  for  verifying  that 
evidence  will  be  formulated. 

For  practices  that  lack  objective  evidence,  a  strategy  for  discovering  that 
evidence  will  be  formulated. 

The  data  collection  plan  is  typically  embodied  in  a  number  of  different 
artifacts  used  during  the  appraisal  process.  The  appraisal  plan  includes 
information  about  the  site,  projects,  and  participants  involved  in  the  appraisal. 
This  is  the  highest  level  of  information  that  helps  document  and  communicate 
the  data  collection  plan.  Detailed  information  on  data  collection  can  be 
recorded  in  work  aids  that  manage  appraisal  data  and  in  the  appraisal 
schedule.  A  record  of  information  needed”  items  is  the  most  detailed 
representation,  while  document  lists,  interview  schedules,  and  the  assignment 
of  PA  mini-teams  help  shape  the  strategy  for  obtaining  the  needed  data. 


•  Determine  participants  for  interviews. 

•  Determine  artifacts  to  be  reviewed. 

•  Determine  presentations/demonstrations  to  be  provided. 

•  Determine  team  roles  and  responsibilities  for  data  collection  activities. 

•  Document  the  data  collection  plan. 


For  every  instantiation  of  every  model  practice,  the  data  collection  plan  must 
specify  how,  when,  and  by  whom  the  objective  evidence  will  be  verified. 

For  instantiations  of  model  practices  that  have  not  been  addressed  in  the 
initial  objective  evidence,  the  data  collection  plan  must  specify  how  the  team 
intends  to  discover  the  presence  or  absence  of  objective  evidence  that 
characterizes  the  extent  of  implementation  for  that  practice. 
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1 .5.2  Prepare  Data  Collection  Plan  (continued) 


Parameters 
and  Limits 
(continued) 


Optional 

Practices 


The  data  collection  plan  (often  documented  in  a  variety  of  artifacts)  includes 

•  assignment  of  PAs  to  team  members 

•  summary  of  initial  objective  evidence  provided  by  the  organization 

•  identification  of  highest  priority  data  needs 

•  initial  allocation  of  data  needs  to  data-gathering  events 

•  identification  of  instruments  to  be  administered 

•  identification  of  participants  to  be  interviewed 

•  interview  schedule,  revised  to  include  more  detail 

•  identification  of  a  starter  set  of  interview  questions 

•  identification  of  documents  still  needed  (if  any) 

•  risks  associated  with  the  sufficiency  of  the  data  and  the  adequacy  of  the 
schedule 


Review  the  status  of  the  objective  evidence  database  with  members  of  the 
appraised  organization  to  elicit  additional  objective  evidence  or  to  expand  on 
the  evidence  available.  This  allows  the  appraisal  team  leader  to  validate  the 
data  collection  plan  to  some  extent. 


Continued  on  next  page 
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1 .5.2  Prepare  Data  Collection  Plan  (continued) 


implementation 

Guidance 


Sources  of  objective  evidence  include  instruments,  documents,  presentations, 
and  interviews  (see  process  2.1,  Examine  Objective  Evidence).  Objective 
evidence  is  differentiated  in  terms  of  different  types  of  PIIs  (direct  artifacts, 
indirect  artifacts,  and  affirmations),  as  described  in  activity  1.4.3,  Obtain 
Initial  Objective  Evidence.  A  combination  of  these  indicator  types  is  required 
for  corroboration  (see  activity  2.2.1,  Verify  Objective  Evidence).  The  data 
collection  status  is  continually  monitored  during  appraisal  activities  (see 
process  2.3,  Document  Objective  Evidence)  to  ensure  that  sufficient  data 
coverage  is  obtained.  These  are  all  key  considerations  that  should  be 
understood  and  accounted  for  in  the  generation  of  the  data  collection  plan. 

Multiple  types  of  interviews  can  be  used  to  obtain  face-to-face  affirmations 
(see  activity  2.1.4,  Examine  Objective  Evidence  from  Interviews): 

•  standard  structured  interviews  scheduled  in  advance  and  using  scripted 
questions 

•  on-call  interviews,  scheduled  in  advance  for  calendar  purposes,  but  held 
only  if  it  is  determined  they  are  necessary 

•  office  hours  interviews,  for  which  interviewees  are  notified  that  they  may 
need  to  be  available  as  a  contingency  during  scheduled  periods 

A  robust  data  collection  plan  will  plan  for  interviews  of  all  three  types.  Start 
with  a  full  set  of  scheduled  interviews  early  in  the  planning  phase,  and 
gradually  add/eliminate/modify  events  as  the  inventory  of  initial  objective 
evidence  indicates  the  need.  The  mini-teams  may  conduct  office  hours 
interviews,  even  during  team  training,  to  more  fully  populate  the  inventory  of 
objective  evidence  prior  to  the  start  of  the  on-site  data  collection  activities. 

Planning  for  document  reviews  should  include  organizational-,  project-,  and 
implementation-level  artifacts,  as  described  in  activity  2.1.3,  Examine 
Objective  Evidence  from  Documents. 

Ultimately,  the  appraisal  team  will  need  to  have  data  on  each  practice  in  the 
CMMI  model,  for  each  organizational  element  in  the  appraisal  scope.  For  PAs 
addressing  practices  implemented  at  the  project/program  level  (e.g..  Project 
Planning),  this  means  that  data  on  each  instantiation  of  the  practice  will  be 
collected.  For  PAs  addressing  practices  implemented  at  the  organization  level 
(e.g..  Organizational  Training),  only  one  instantiation  of  each  practice  may  be 
needed,  depending  on  the  way  the  organization  chooses  to  implement  such 
practices. 
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1.5.2  Prepare  Data  Collection  Plan  (continued) 


Implementation  The  results  of  the  analysis  of  initial  objective  evidence  are  used  to  determine 
Guidance  which  practices  are  not  already  covered  with  objective  evidence.  Practices  for 

(continued)  which  no  initial  objective  evidence  has  been  provided  should  be  identified  as 

high-risk  areas  for  the  team  to  address  immediately.  The  schedule  for  data 
collection  may  need  to  change  dramatically  if  the  team  is  unable  to  find 
relevant  data  for  these  areas  in  short  order.  In  the  case  of  practices  for  which 
data  are  available  in  the  initial  objective  evidence,  the  team  members  assigned 
to  the  PAs  plan  the  strategy  for  verifying  the  implementation  of  each  of  the 
practices  through  review  of  the  named  documents,  interviews  with  the  people 
who  play  the  named  roles,  or  other  data  collection  events.  Artifacts  used  to 
manage  data  collection  events  are  populated  with  the  current  understanding  of 
the  planned  data  collection  events,  as  follows: 

•  The  schedule  for  interviews  is  finalized,  so  participants  can  be  informed 
of  the  expectations  for  their  participation  as  interviewees. 

•  The  list  of  documents  on  hand  (or  accessible  electronically)  is  finalized, 
so  that  the  team  members  know  what  is  and  is  not  available  for  document 
review. 

•  A  preliminary  allocation  of  practices  to  be  covered  in  each  of  the 
scheduled  interviews  is  documented. 

•  A  list  of  needed  documents  (not  yet  available  to  the  team)  is  generated,  if 
there  are  any  known  needs  for  documents  at  this  point. 


♦ 


CMU/SEI-2001-HB-001 


Page  II-67 


1 .5.3  Replan  Data  Collection 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  data  collection  plan  is  updated  as  required  during  the  conduct  of  the 
readiness  review  or  during  the  appraisal  itself  as  objective  evidence  is  found, 
or  as  new  sources  of  information  are  uncovered.  The  activity  described  in  this 
section  refers  to  a  more  substantial  change  in  the  plan,  which  is  expected  to  be 
a  rare  occurrence  in  practice.  If  during  the  conduct  of  an  appraisal,  the  team 
discovers  that  their  assumptions  about  the  availability  of  objective  evidence 
are  substantially  incorrect,  the  appraisal  team  leader  may  renegotiate  the 
appraisal  plan  with  the  sponsor. 


•  Review  the  current  inventory  of  objective  evidence  and  determine  model 
practices  for  which  the  objective  evidence  is  inadequate  relative  to  the 
appraisal  plan. 

•  Revise  the  data  collection  plan  as  necessary  based  on  the  appraisal  status 
and  availability  of  objective  evidence. 

•  Renegotiate  the  appraisal  plan  with  the  sponsor  if  the  appraisal  cannot 
proceed  as  planned. 


This  activity  is  not  a  substitute  for  tactical  decisions  about  where  and  how  to 
find  objective  evidence.  The  intent  of  this  activity  is  to  respond  to  a  major  gap 
between  expected  data  and  actual  data. 

gaps  between  expected  and  actual  data  may  occur,  for  example,  as  a 
result  of  the  following: 

•  inaccurate  assumptions  about  the  availability  of  objective  evidence 

•  content  of  artifacts  or  information  from  interviews  not  providing 
significant  amounts  of  the  information  required  and  other  somces  not 
being  planned 

•  unexpected  absence  of  multiple  key  interviewees 

•  unanticipated  delays  in  the  implementation  of  new  processes 

•  major  customer-driven  emergencies  for  one  or  more  of  the  sampled 
projects 
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1 .5.3  Replan  Data  Collection  (continued) 


Optional  Risk  analysis  can  be  conducted  during  early  planning  activities  to  establish 

Practices  thresholds  and  limits  for  the  amount  of  missing  objective  evidence  that  will 

trigger  this  activity.  This  enables  the  appraisal  team  leader  to  state,  in 
advance,  the  conditions  under  which  the  team  and  the  sponsor  must 
renegotiate  the  appraisal  plan. 

Contingency  planning  done  in  advance  to  identify  ways  of  overcoming  issues 
associated  with  missing  objective  evidence  could  include 

•  an  alternate  (fall-back)  schedule  for  the  appraisal 

•  staffing  to  conduct  a  “crash  data  collection”  activity 

•  reducing  the  scope  of  the  appraisal  (e.g.,  appraising  fewer  PAs,  limiting 
the  extent  of  the  organizational  unit  appraised) 


Implementation  This  activity  serves  as  a  “pressure  valve”  of  sorts  for  the  appraisal.  The 
Guidance  pressure  to  perform  the  appraisal  under  unrealistic  conditions  can  lead  to  a 

severe  degradation  in  the  quality  of  the  appraisal  outputs.  Carefully  planning 
for  contingencies  and  communicating  them  to  the  sponsor  help  to  protect  the 
standards  that  must  be  met  in  the  performance  of  an  appraisal.  Clearly 
documenting  the  data  collection  plan,  and  regularly  monitoring  the 
availability  of  data  compared  to  that  plan,  support  effective  risk  mitigation. 

When  this  activity  must  be  employed  to  recover  from  an  unrealistic 
expectation,  the  documentation  reflecting  the  assumptions  made  during 
planning,  as  well  as  concrete  facts  about  what  is  or  is  not  available,  are  used 
to  renegotiate  with  the  appraisal  sponsor.  This  is  one  of  the  reasons  why  a 
detailed  appraisal  plan,  with  the  sponsor’s  signature,  is  a  required  artifact  for 
the  conduct  of  a  SCAMPI  appraisal. 
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2.1  Examine  Objective  Evidence 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 

Outcome 


Exit  Criteria 


Collect  information  about  the  practices  implemented  in  the  organization  and 
relate  the  resultant  data  to  the  reference  model.  Perform  the  activity  in 
accordance  with  the  data  collection  plan.  Take  corrective  actions  and  revise 
the  data  collection  plan  as  needed. 


•  Data  collection  has  been  planned. 

•  The  sponsor  has  approved  the  appraisal  plan. 

•  The  appraisal  team  is  trained  and  is  familiar  with  the  appraisal  plan. 

•  Participants  have  been  informed  about  the  appraisal  process  and  their 
roles  in  it. 


Appraisal  data 

initial  objective  evidence 
documents 

documented  practice  implementation  gaps,  if  any 

feedback  on  preliminary  findings  (if  that  point  in  the  timeline  has 

been  reached) 

Data  collection  plan 
appraisal  schedule 
interview  schedule 
document  list 
new  interview  questions 


2.1.1  Examine  Objective  Evidence  from  Instruments 

2. 1 .2  Examine  Objective  Evidence  from  Presentations 

2.1.3  Examine  Objective  Evidence  from  Documents 

2. 1 .4  Examine  Objective  Evidence  from  Interviews 


•  Updated  appraisal  data 

•  Updated  data  collection  plan 


After  the  final  iteration  of  this  process,  the  team  has  sufficient  data  to  create 
appraisal  findings  and  to  make  judgments  about  the  implementation  of 
practices,  as  well  as  the  satisfaction  of  specific  and  generic  goals. 


The  coverage  of  the  reference  model  and  the  organizational  scope  has  been 
achieved,  and  the  team  is  ready  to  produce  the  appraisal  outputs. 
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2.1  Examine  Objective  Evidence  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification  and 
Validation 


Records 


Tailoring 


The  efficient  collection  of  objective  evidence  results  from  carefully  creating 
and  executing  the  data  collection  plan.  Effective  contingency  planning  and  the 
use  of  work  aids  to  monitor  progress  are  key  points  to  consider.  The  team 
must  be  able  to  focus  on  examining  the  most  relevant  information  available, 
rather  than  be  distracted  by  a  mission  to  root  out  new  evidence. 


Wall  charts  and  other  visual  aids  are  often  used  to  display  the  results  of  data 
collection  activities.  Electronic  tools  are  prevalent  among  experienced  Lead 
Appraisers,  and  can  be  very  effective  for  continually  monitoring  and  updating 
the  inventory  of  objective  evidence. 


Tracking  the  actual  coverage  obtained,  as  compared  to  the  planned  coverage, 
in  each  data  collection  activity  facilitates  timely  corrective  actions  where  they 
are  needed.  The  most  critical  resource  during  an  appraisal  is  time.  Using  a 
timekeeper  during  data  collection  and  verification  activities  provides  feedback 
on  team  performance.  Recording  the  actual  duration  of  planned  events  helps 
the  team  in  taking  actions  to  recover  from  unexpected  events. 


The  appraisal  method  provides  detailed  verification  and  validation  procedures 
for  objective  evidence.  They  are  described  in  process  7,  Verify  and  Validate 
Objective  Evidence. 


Work  aids  used  to  record  and  track  the  progress  of  data  collection  activities 
are  retained  for  traceability  and  provide  an  important  input  to  a  final  report 
describing  the  appraisal,  if  the  sponsor  has  requested  a  final  report.  The 
duration  and  effort  required  for  specific  data  collection  events  can  be  recorded 
to  provide  useful  historical  data  for  planning  subsequent  appraisals. 


The  method  is  flexible  in  terms  of  the  use  of  customized  data  collection 
instruments,  presentations,  document  reviews,  and  interviews.  Specialized 
forms  of  these  data  collection  methods  can  be  constructed  to  meet  the 
objectives  of  the  appraisal.  For  example,  an  organization-specific 
questionnaire  could  be  used  that  contains  local  jargon  rather  than  a 
standardized  questionnaire.  Standardized  presentations  can  be  employed  to 
provide  the  team  with  an  “inbrief’  at  the  start  of  the  appraisal.  The  method 
also  provides  flexibility  in  choosing  the  number,  duration,  style,  and  make-up 
of  interview  sessions  within  specified  boundaries. 
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2.1  Examine  Objective  Evidence  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  activities  that  provide  the  team  with  data  needed  to  produce  reliable 
appraisal  outputs  are  perhaps  the  most  visible  part  of  the  appraisal  process 
from  the  perspective  of  the  appraised  organization.  For  this  reason,  SCAMPI 
places  a  heavy  emphasis  on  methodically  planning  and  tracking  the  data 
collected  during  an  appraisal.  The  initial  objective  evidence  collected  early  in 
the  process  allows  team  members  to  analyze  the  state  of  information  available 
at  the  earliest  stages  of  the  appraisal  and  narrow  their  search  for  new 
information.  This  early  work  serves  to  facilitate  an  efficient  use  of  time.  An 
explicit  understanding  of  what  information  is  needed  and  how  that 
information  will  be  used  therefore  drives  the  activities  associated  with  this 
process. 


The  members  of  the  team  continually  manage  the  data  collected  previously 
and  target  new  data  collection  activities  to  fill  known  information  needs. 
Instruments  tend  to  be  used  early  in  the  appraisal  process,  and  often  provide 
leads  to  be  pursued  through  other  data  collection  activities,  in  addition  to 
affirmations  of  implemented  practices.  Presentations  are  sometimes  used  to 
provide  a  flexible  forum  where  members  of  the  organization  can  explain 
important  information  about  the  practices  implemented  in  the  organization. 
Documents  provide  the  most  explicit  and  lasting  representation  of  practice 
implementation  in  the  organization,  and  the  team  uses  them  to  understand 
how  practices  in  the  CMMI  model  are  implemented.  Finally,  interviews  are 
used  as  the  most  dynamic  data  collection  technique,  allowing  for  branching 
among  related  topics  and  the  explanation  of  contextual  information  that 
affects  the  implementation  of  practices  as  well  as  alternative  practices. 

The  appraisal  activities  conducted  for  each  of  these  data  collection  sources  are 
similar: 

•  Determine  if  the  information  obtained  is  acceptable  as  objective  evidence. 

•  Relate  the  objective  evidence  to  corresponding  practices  in  the  appraisal 
reference  model. 

•  Relate  the  objective  evidence  to  the  appropriate  part  of  the  appraised 
organizational  unit  (i.e.,  the  instantiation). 
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2.1.1  Examine  Objective  Evidence  from  Instruments 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Instruments  provided  by  the  organizational  unit  are  reviewed  to  obtain 
objective  evidence  reflecting  the  organization’s  implementation  of  model 
practices.  Instruments  include  questionnaires,  surveys,  and  other  written 
information  that  indicates  practice  implementation. 

This  activity  builds  upon  the  inventory  of  objective  evidence  that  was 
developed  during  appraisal  planning  and  preparation.  The  appraisal  team 
considers  the  information  contained  in  the  instruments  and  determines  if  it  is 
accurate,  consistent,  and  relevant  to  the  scope  of  the  reference  model  being 
examined. 

Objective  evidence  obtained  from  instruments,  and  from  other  sources,  is 
documented  in  process  2.3,  Document  Objective  Evidence,  and  verified  in 
process  2.2,  Verify  and  Validate  Objective  Evidence. 


•  Review  information  obtained  from  instruments  and  determine  if  it  is 
acceptable  as  objective  evidence. 

•  Determine  the  model  practices  corresponding  to  the  objective  evidence 
obtained  from  instruments. 

•  Determine  the  portions  of  the  organizational  unit  that  correspond  to  the 
objective  evidence  obtained  from  instruments. 


At  least  one  instrument  must  be  used  during  the  conduct  of  the  appraisal. 
(Refer  to  1.4.2,  Administer  Instruments,  for  a  description  of  instruments.) 
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2.1 .1  Examine  Objective  Evidence  from  Instruments  (continued) 


Optional  Summaries  of  practice  implementation  data  (collected  via  instruments)  for  a 

Practicos  group  of  projects  in  an  organization  may  be  useful  during  the  selection  of  the 

projects  used  to  represent  the  organizational  unit. 

•  The  use  of  legacy  processes  (versus  newly  deployed  processes)  can  be 
flagged  using  the  responses  to  these  instruments.  This  can  support  the 
inclusion  or  exclusion  of  projects  using  various  versions  of  the 
organization’s  set  of  standard  processes. 

•  This  can  also  help  flag  situations  where  projects  have  not  yet  reached  a 
particular  point  in  the  life  cycle,  allowing  the  appraisal  team  leader  to 
avoid  the  anomalous  situation  where  none  of  the  sampled  projects  has  yet 
reached  the  point  where  a  practice  under  investigation  has  been 
implemented. 

Create  and  administer  a  specialized  questionnaire  that  is  tailored  to  the 
characteristics  of  the  organization,  or  the  objectives  of  the  appraisal. 


Implementation  The  use  of  instruments  to  gather  written  information  from  members  of  the 
Guidance  organization  provides  a  relatively  low-cost  data  collection  technique,  when 

done  well.  Data  of  this  type  tend  to  be  most  useful  when  provided  early  in  the 
appraisal  conduct,  and  can  lead  to  valuable  insights  about  where  data  may  be 
sought  during  subsequent  data  collection  events. 

The  most  common  instrument  used  is  the  organization’s  PII  database,  which 
provides  traceability  of  reference  model  practices  to  the  processes  and  work 
products  implemented  within  the  organization.  Where  organizations  have  not 
yet  implemented  this  asset,  a  questionnaire  can  be  used  to  gather  closed-ended 
responses  and  comments  about  the  implementation  of  each  model  practice  in 
each  sampled  project  in  the  organizational  unit. 

It  is  also  the  responsibility  of  the  appraisal  team  leader  to  prevent  duplicate 
data  entry  on  multiple  instruments.  No  organization  should  be  asked  to 
provide  the  same  information  in  two  (or  more)  formats. 
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2.1.2  Examine  Objective  Evidence  from  Presentations 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Demonstrations  of  on-line  tools,  or  libraries  to  be  accessed  by  the  appraisal 
team,  are  often  the  best  way  for  members  of  the  team  to  find  the  data  and 
information  they  need.  The  history  of  process  improvement  in  the 
organization  or  the  status  of  current  improvement  projects  can  sometimes  be 
best  conveyed  to  the  appraisal  team  in  the  form  of  a  presentation.  While  the 
amount  of  data  to  be  collected  using  presentations  will  be  minimal,  the  ability 
to  receive  information  and  ask  questions  in  real  time  makes  this  a  valuable 
data  collection  technique. 

Objective  evidence  obtained  from  presentations,  and  from  other  sources,  is 
documented  in  process  2.3,  Document  Objective  Evidence,  and  verified  in 
process  2.2,  Verify  and  Validate  Objective  Evidence. 


•  Receive  presentations,  if  applicable,  from  the  organizational  unit. 

•  Review  information  obtained  from  presentations  and  determine  if  it  is 
acceptable  as  objective  evidence. 

•  Determine  the  model  practices  corresponding  to  the  objective  evidence 
obtained  from  presentations. 

•  Determine  the  portions  of  the  organizational  unit  that  correspond  to  the 
objective  evidence  obtained  from  presentations. 


There  is  no  requirement  for  one  or  more  presentations  to  be  included  in  the 
data  collection  plan.  The  team  must  permit  presentations  of  information  by 
knowledgeable  members  of  the  organization.  Presentations  may  or  may  not  be 
“required”  by  the  team,  depending  on  the  appraisal  usage  mode  and  the 
appraisal  objectives. 

It  is  not  necessary  that  all  team  members  be  present  at  every  presentation, 
though  it  may  be  advantageous.  A  minimum  of  two  team  members  must  be 
present  in  order  to  consider  any  presentation  a  valid  data  collection  session. 

Team  members  take  notes  during  presentations  to  document  information  for 
later  use,  as  described  in  activity  2.3.1,  Take/Review/Tag  Notes. 


Continued  on  next  page 
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2.1 .2  Examine  Objective  Evidence  from  Presentations  (continued) 


Optional  Allow  the  organization  to  provide  presentations  or  demonstrations  of  tools,  as 

Practices  a  means  of  providing  objective  evidence  about  the  implementation  of  model 

practices. 

Establish  a  standardized  boilerplate  for  the  organizational  unit,  or  projects 
within  the  organizational  unit,  to  use  in  orienting  the  appraisal  team. 


Implementation  Presentations  about  the  history  of  process  improvement  in  an  organization  can 
Guidance  be  very  revealing,  and  can  help  to  shape  the  emphasis  for  further  data 

collection. 

Demonstrations  of  tools  supporting  the  process  infrastructure  are  sometimes 
the  most  convenient  means  of  communicating  objective  evidence.  Tools  that 
are  commonly  demonstrated  include 

•  requirements  management  and  traceability  tools 

•  configuration  management  library 

•  metrics  database 

•  process  asset  library  and  tools 

•  process-related  Web  pages 

•  computer-based  training  courses  or  training  repositories 

•  risk  management  databases 

A  configuration  management  library  often  embodies  the  process  by  which 
engineers  manage  configurations.  These  engineers  may  take  for  granted  that 
certain  standards  are  enforced  through  the  tool  and  be  unable  to  explain  what 
those  standards  are  in  the  abstract. 

An  organization’s  metrics  database  can  often  embody  the  analytical 
techniques  in  use,  as  well  as  the  communication  channels  that  are  supported 
across  the  organizational  unit. 
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2.1.3  Examine  Objective  Evidence  from  Documents 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


A  substantial  portion  of  the  data  used  by  appraisal  team  members  is  derived 
from  documents  they  review.  Most  of  the  direct  artifacts  used  as  indicators  of 
practice  implementation  are  documents.  Document  review  is  an  effective 
means  to  gain  detailed  insight  about  the  practices  in  use  in  the  organization. 
However,  without  a  clear  focus  on  the  data  being  sought,  document  review 
can  consume  a  great  deal  of  time  as  team  members  sometimes  attempt  to  read 
everything  in  hopes  that  something  useful  will  be  discovered. 

Objective  evidence  C'btained  from  documents,  and  from  other  sources,  is 
documented  in  process  2.3,  Document  Objective  Evidence,  and  verified  in 
process  2.2,  Verify  and  Validate  Objective  Evidence. 


•  Establish  and  maintain  a  catalogue  of  documents  used  as  a  source  of 
Objective  Evidence. 

•  Review  information  obtained  from  documents  and  determine  if  it  is 
acceptable  as  objective  evidence. 

•  Determine  the  model  practices  corresponding  to  the  objective  evidence 
obtained  from  documents. 

•  Determine  the  portions  of  the  organizational  unit  that  correspond  to  the 
objective  evidence  obtained  from  documents. 


All  SCAMPI  appraisals  must  use  documents  as  a  source  of  information  on  the 
extent  to  which  practices  have  been  implemented  in  the  organizational  unit 
and  within  the  sampled  projects. 

The  catalogue  should  be  sufficient  to  summarize  the  documentation  objective 
evidence  used  as  a  basis  for  appraisal  ratings  generated,  as  required  by  the 
Appraisal  Record  described  in  activity  3.2.2,  Generate  Appraisal  Record. 
Much  of  the  catalogue  contents  can  be  obtained  from  the  mapping  data  or 
instruments  obtained  from  the  organizational  unit,  such  as  the  PII  database,  or 
questionnaires.  The  catalogue  can  be  used  to  maintain  a  list  of  documents 
reviewed  or  additional  documentation  requested  from  the  organizational  unit. 


Continued  on  next  page 
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2.1 .3  Examine  Objective  Evidence  from  Documents  (continued) 

Optional  For  organizations  with  substantial  intranets  containing  Web-based  document 

Practices  libraries,  a  member  of  the  organization  familiar  with  the  document  library 

should  provide  a  demonstration  of  the  Web-based  tools.  Links  to  other 
documents  and  other  features  of  the  Web-based  document  library  must  be 
tested  prior  to  the  team’s  use  during  the  appraisal. 

Implementation  One  or  more  team  members  will  seek  data  for  every  practice  in  the  model 
Guidance  scope  of  the  appraisal  through  document  review.  This  does  not  require  a 

document  for  every  practice,  as  any  given  document  is  likely  to  provide  data 
relevant  to  multiple  practices.  To  the  greatest  extent  possible,  the  location  of 
documented  evidence  relating  to  every  practice  should  be  recorded  in  advance 
of  the  team’s  arrival  at  the  site  where  the  appraisal  will  occur.  Organizations 
with  established  improvement  infrastructures  typically  maintain  this  type  of 
information  in  order  to  track  their  improvement  efforts  against  the  model. 
Where  this  infomnation  is  incomplete,  the  team  will  be  forced  to  discover  the 
linkages  between  the  CMMI  model  and  the  organization’s  implemented 
practices,  and  will  therefore  require  more  time  to  perform  the  appraisal. 

Implementation  Documents  reviewed  during  an  appraisal  can  be  classified  into  three  different 
Guidance  levels:  organization,  project,  and  implementation. 

By  providing  further  insight  into  the  policies  and  procedures  that  guide  the 
organization’s  processes,  organization-level  documents  sometimes  help  the 
team  to  eliminate  the  need  for  a  question  during  an  interview  or  sharpen  the 
focus  for  a  question.  Review  of  these  documents  provides  a  context  for 
understanding  the  expectations  placed  on  projects  within  the  organization. 

Through  the  review  of  project-level  documents,  team  members  gain  further 
insight  into  each  scheduled  interviewee’s  role  in  the  project  they  support  as 
well  as  the  terminology  generally  accepted  within  the  organization  or  project. 
This  may  lead  to  the  refinement  or  modification  of  interview  questions. 

The  team  typically  reviews  implementation-level  documents  to  validate 
information  gathered  from  other  sources,  such  as  interviews  or  higher-level 
documents.  Documents  on  this  level  provide  an  audit  trail  of  the  processes 
used  and  the  work  performed.  The  review  of  these  documents  frequently 
provides  verification  of  practices  found  in  organization-  and  project-level 
documents. 

♦ 


Three  Levels  of 
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2.1.4  Examine  Objective  Evidence  from  Interviews 


Activity 

Description 


Required 

Practices 


Interviews  are  used  to  obtain  face-to-face  affirmations  relating  to  the 
implementation  of  processes  at  the  organizational  and  project  levels. 
Interviews  are  held  with  managers  and  practitioners  responsible  for  the  work 
being  performed.  The  appraisal  team  uses  interviews  to  understand  how  the 
processes  are  implemented  and  to  probe  areas  where  additional  coverage  of 
model  practices  is  needed. 

Interviews  are  a  required  and  necessary  component  of  a  SCAMPI  appraisal,  in 
all  usage  modes.  The  criteria  for  the  amount  of  face-to-face  affirmation 
objective  evidence  that  must  be  collected  are  described  in  activity  2.2.1, 
Verify  Objective  Evidence.  This  drives  the  development  of  the  initial 
interviewing  strategy  in  the  data  collection  plan  described  in  activity  1.5.2, 
Prepare  Data  Collection  Plan.  A  variety  of  interviewing  techniques  are 
available,  and  the  appraisal  team  leader  works  with  the  team  to  schedule  the 
most  appropriate  interview  types  for  the  situation. 

As  objective  evidence  is  gathered  throughout  the  appraisal,  the  data  collection 
plan  is  revised  as  necessary.  By  using  focused  investigation  techniques,  the 
need  for  interviews  may  be  either  increased  or  diminished,  as  long  as  the 
criteria  for  face-to-face  affirmations  are  satisfied. 

Objective  evidence  obtained  from  interviews,  and  from  other  sources,  is 
documented  in  process  2.3,  Document  Objective  Evidence,  and  verified  in 
process  2.2,  Verify  and  Validate  Objective  Evidence. 


•  Refine  the  data  collection  plan  to  determine  the  objective  evidence  that 
must  be  obtained  from  interview  participants. 

•  Review  information  obtained  from  interviews  and  determine  if  it  is 
acceptable  as  objective  evidence. 

•  Determine  the  model  practices  corresponding  to  the  objective  evidence 
obtained  from  interviews. 

•  Determine  the  portions  of  the  organizational  unit  that  correspond  to  the 
objective  evidence  obtained  from  interviews. 
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2.1 .4  Examine  Objective  Evidence  from  Interviews  (continued) 


Parameters 
and  Limits 


Optional 

Practices 


All  SCAMPI  appraisals  must  use  interviews  as  a  source  of  information  on  the 
extent  to  which  practices  have  been  implemented  in  the  organizational  unit 
and  within  the  sampled  projects. 

All  interviews  must  include  at  least  two  members  of  the  appraisal  team 
designated  by  the  appraisal  team  leader. 

Full  coverage  of  the  CMMI  model,  the  organizational  unit,  and  the 
organization’s  life  cycle(s)  must  be  achieved  with  the  objective  evidence 
considered  by  the  team.  Therefore  the  pool  of  potential  interviewees  must 
cover  all  elements  of  the  process  in  use  in  the  organizational  unit. 

Project  and/or  program  management  personnel  are  typically  interviewed 
individually,  or  grouped  according  to  project.  The  focus  of  the  discussion  in 
these  interviews  will  therefore  be  scoped  to  a  particular  project,  rather  than 
across  the  sampled  projects. 

Functional  Area  Representatives  (FARs)  are  typically  interviewed  in  a  group, 
sampling  across  the  projects  within  the  organizational  unit.  The  focus  of  the 
discussion  in  these  interviews  will  therefore  be  scoped  to  a  particular  set  of 
practices,  used  across  the  projects. 

The  rules  of  confidentiality  and  the  expected  use  of  appraisal  data  must  be 
communicated  to  every  interviewee. 


Request  that  interviewees  bring  a  document  or  other  artifact  with  them  to  their 
interviews  for  a  “show-and-tell”  style  interview. 

Use  video/teleconference  technology  to  conduct  interviews  at  a  distance. 
Appraisers  are  cautioned  not  to  rely  too  heavily  on  this  method.  If  substantial 
portions  of  the  interview  data  are  gathered  using  this  technology,  it  may  tend 
to  limit  the  amount  of  information  collected. 


Continued  on  next  page 
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2.1 .4  Examine  Objective  Evidence  from  Interviews  (continued) 


Implementation 

Guidance 


Interviews  provide  the  most  flexible  source  of  detailed  data.  Face-to-face 
interaction  with  people  who  enact  the  practices  being  investigated  allows  the 
team  to  seek  detailed  information  and  to  understand  the  interrelationships 
among  various  practices.  Detailed  information  to  address  specific  data 
collection  needs  can  be  sought  and  verified  in  real  time. 

It  is  important  to  avoid  sampling  interviewees  for  a  session  such  that  two 
people  in  the  same  reporting  chain  (e.g.,  a  superior  and  one  of  his  or  her  direct 
reports)  are  in  the  same  interview  session.  This  applies  to  members  of  the 
appraisal  team  as  well.  People  who  have  this  type  of  relationship  with  one 
another  may  be  uncomfortable  with  the  expectation  for  them  to  be  completely 
candid  during  the  interview. 

Samples  of  interviewees  are  typically  grouped  into  categories  that  roughly 
correspond  to  life-cycle  phases,  engineering  disciplines,  organizational 
groupings,  and/or  PA  affinities.  As  stated  previously,  interviews  of 
project/program  management  personnel  are  typically  grouped  by  project, 
while  FARs  sampled  for  a  given  interview  come  from  across  the 
organizational  unit. 

There  are  three  basic  forms  of  interviews  used  in  SCAMPI.  They  are 
described  below. 
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2.1 .4  Examine  Objective  Evidence  from  Interviews  (continued) 


Implementation  The  most  structured  approach  is  the  standard  interview,  which  is  scheduled  in 
Guidance  advance  and  employs  a  series  of  scripted  questions.  Each  standard  interview 

typically  involves  interviewees  with  similar  responsibilities  in  the 
Standard  organization  (e.g..  Quality  Assurance  personnel.  Systems  Engineers,  or 

Interviews  Middle  Managers).  The  schedule  and  location  of  each  interview  session  is 

communicated  to  the  interviewees  well  in  advance.  Questions  intended  to 
elicit  data  about  particular  practices  are  prepared  and  reviewed  in  advance, 
and  the  team  follows  a  defined  process  for  conducting  the  session.  The  entire 
team  is  present  for  these  interviews.  Responsibility  for  tracking  the  coverage 
of  individual  PAs  is  typically  assigned  to  team  members.  A  single  questioner 
may  lead  the  interview,  with  the  rest  of  the  team  listening  and  taking  notes,  or 
the  responsibility  for  asking  questions  may  be  distributed  among  the  team 
members.  In  any  case,  it  is  expected  that  all  team  members  who  are  not  asking 
questions  listen  and  take  notes  for  all  questions. 

A  set  of  planned  interviews  will  be  defined  during  appraisal  planning.  As  the 
appraisal  progresses  and  the  objective  evidence  accumulates,  the  team  may 
find  it  convenient  to  cancel  one  or  more  of  these  interviews  to  use  the  time  for 
other  activities.  Such  changes  in  the  data  collection  plan  are  made  in  a  way 
that  does  not  violate  the  coverage  criteria  described  in  process  2.2,  Verify  and 
Validate  Objective  Evidence. 


A  more  flexible  approach  to  scheduling  interviews  is  available  in  the  form  of 
on-call  interviews,  a  variant  of  the  standard  interview.  Prospective 
interviewees  are  identified  and  notified  in  advance,  just  as  described  above. 
However,  the  interviews  are  only  held  if  team  members  decide  that  there  is  a 
need  and  that  the  time  will  be  well  spent.  The  prospective  interviewees  are 
therefore  asked  to  block  a  period  of  time  for  such  a  contingency,  and  are 
informed  the  day  before  the  scheduled  time  as  to  whether  or  not  the  interview 
session  will  actually  happen.  These  interviews  need  not  include  the  entire 
appraisal  team,  thus  permitting  parallel  sessions  with  different  interviewees. 
However,  at  least  two  members  of  the  appraisal  team  (selected  by  the 
appraisal  team  leader)  must  participate. 


Implementation 

Guidance 

On-Call 

Interviews 
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2.1.4  Examine  Objective  Evidence  from  interviews  (continued) 


Implementation  Finally,  office  hours  interviews  represent  an  agreement  for  availability  that 
Guidance  permits  pairs  of  team  members  to  visit  interviewees  at  their  desks,  cubicles,  or 
offices.  As  with  the  on-call  interviews,  the  prospective  interviewees  block  a 
Office  Hours  specific  time  period  to  be  available  on  a  contingency  basis.  It  is  expected  that 

Interviews  most  prospective  interviewees  will  be  able  to  continue  with  their  daily  work 

and  accommodate  an  interruption  if  the  team  needs  to  speak  with  them.  Here 
again,  only  if  specific  data  needs  are  identified  will  the  interview  occur.  The 
interviewees  should  be  informed  that  they  may  receive  only  limited  advanced 
notice  for  these  interviews,  although  confirming  the  interview  at  least  a  day  in 
advance  is  a  courtesy  that  should  be  offered  whenever  possible. 
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2.2  Verify  and  Validate  Objective  Evidence 


Purpose  Verify  the  implementation  of  the  organization’s  practices  for  each 

instantiation,  and  validate  the  preliminary  findings,  describing  gaps  in  the 
implementation  of  model  practices.  Each  implementation  of  each  practice  is 
verified  so  that  it  may  be  compared  to  the  practices  of  the  CMMI  model,  and 
the  team  characterizes  the  extent  to  which  the  practices  in  the  model  are 
implemented.  Gaps  in  practice  implementation  are  captured  and  validated 
with  members  of  the  organization.  Exemplary  implementations  of  model 
practices  may  be  highlighted  as  strengths  to  be  included  in  appraisal  outputs. 

Entry  Criteria  Objective  evidence  has  been  collected  about  the  implementation  of  practices 
in  the  organization.  Gaps  in  the  implementation  of  model  practices  have  been 
identified,  and  the  team  is  ready  to  characterize  the  extent  to  which  model 
practices  (or  acceptable  alternatives  to  those  practices)  have  been 
implemented.  Descriptions  of  practice  implementation  gaps  at  the  level  of  the 
organizational  unit  have  been  crafted  for  validation. 

Inputs  •  Appraisal  plan,  with  schedule  and  participants  for  data  validation  activities 

•  Data  on  practice  implementation,  and  strength/weakness  statements 

•  Data  collection  plan,  specifying  any  additional  information  needed 

Activities  2.2. 1  Verify  Objective  Evidence 

2.2.2  Characterize  Implementation  of  Model  Practices 

2.2.3  Validate  Practice  Implementation  Gaps 

Outputs  •  Updated  appraisal  data 

notes 

strength/weakness  statements 
annotated  worksheets 

•  Updated  appraisal  artifacts 

preliminary  Endings 
revised  data  collection  plan 
requests  for  additional  data 

JO 

Outcome  The  team’s  confidence  in  the  material  that  will  form  the  basis  for  appraisal  io 

outputs  is  increased,  and  the  process  of  transferring  ownership  of  these  results 
has  been  started.  Any  critical  deficiencies  in  the  data  on  hand  have  been 
identified  and  actions  to  resolve  these  issues  have  been  initiated. 

Exit  Criteria  The  team  has  recorded  data  on  the  implementation  of  practices  in  the 
organization,  and  characterized  the  extent  to  which  practices  in  the  model  are 
implemented.  In  addition,  strength  and  weakness  statements  have  been 
validated  with  members  of  the  organization  who  provided  appraisal  data. 
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2.2  Verify  and  Validate  Objective  Evidence  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification  and 
Validation 


Records 


Tailoring 


This  activity  spans  a  number  of  distinct  events  in  the  appraisal  method  that 
together  accomplish  the  same  goal — ensuring  the  validity  of  the  appraisal  data 
and  associated  outputs.  Managing  the  interaction  with  people  outside  of  the 
team  is  a  vitally  important  process  to  ensure  that  the  results  will  be  accurate. 


Facilitation  techniques  to  guide  the  team  through  difficult  decisions  are 
important  during  this  activity  (as  they  are  during  the  Rating  activity  as  well). 
Techniques  to  enhance  the  credibility  of  the  preliminary  findings  are  also 
important.  Using  a  flip  chart  or  note-taker  during  the  presentation  of 
preliminary  findings  is  often  effective  for  instilling  confidence  among 
audience  members. 


Planned  versus  actual  effort  expended  for  this  activity  (as  with  all  activities) 
will  assist  in  monitoring  progress  as  well  as  planning  subsequent  appraisals. 
Gauging  the  level  of  acceptance  for  preliminary  findings  can  be  facilitated  by 
computing  the  percentage  of  findings  adjusted  based  on  feedback,  then 
comparing  this  value  with  past  experience. 


The  attendees  of  preliminary  findings  presentations  are  likely  to  express 
agreement  and/or  discuss  issues  with  the  data  being  validated.  The  appraisal 
team  leader  needs  to  ensure  active  participation  in  these  activities  as  a  way  of 
verifying  that  the  verification  and  validation  process  is  working  as  intended. 
The  actions  taken  following  the  appraisal  will  provide  feedback  to  help 
validate  that  this  activity  was  successful. 


Characterizations  of  practice  implementation,  strength/weakness  statements 
and  changes  made  based  on  feedback  will  be  recorded  for  subsequent  use  by 
the  team. 


Validating  data  is  required,  but  a  variety  of  choices  for  orchestrating  this 
process  are  available.  The  most  common  approach  is  the  preliminary  findings 
presentation.  The  use  of  an  instrument  or  a  more  targeted  focus-group 
approach  to  validate  statements  of  practice  implementation  gaps  is  permitted. 
Also,  the  relative  emphasis  of  mini-team-based  verification  and  verification 
carried  out  by  the  team  as  a  whole  can  be  adjusted  to  meet  the  skills  and 
preferences  of  the  team  at  hand. 
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2.2  Verify  and  Validate  Objective  Evidence  (continued) 

Interfaces  During  the  conduct  of  an  appraisal,  the  team  must  gather  and  analyze  a  great 

with  Other  deal  of  detailed  information.  Processes  described  earlier  in  this  document 

Processes  clarify  how  data  are  gathered  and  examined.  The  process  described  here  is 

focused  on  understanding  the  information  revealed  by  the  data.  The  processes 
described  after  this  one  are  focused  on  carefully  recording  important 
information  and  making  reliable  and  valid  rating  judgments  based  on  the 
verified  and  validated  data. 

Summary  of  The  initial  objective  evidence  provided  by  the  organization  is  used  to 
Activities  understand  how  practices  are  intended  to  be  implemented.  Members  of  the 

appraisal  team  then  seek  information  to  confirm  that  the  intended  practices  are 
indeed  implemented.  This  first  validation  activity  (2.2.1)  may  reveal  gaps  in 
the  actual  implementation  that  are  not  apparent  in  the  initial  objective 
evidence  provided  by  the  organization.  The  next  verification  activity  (2.2.2) 
then  compares  the  implemented  practices  to  the  practices  in  the  CMMI  model. 
This  activity  may  also  reveal  gaps  in  the  implementation(s)  that  will  later  bear 
on  the  ratings  assigned  by  the  team.  Standard  characterizations  to  capture  the 
extent  of  practice  implementation,  first  at  the  project  level  and  then  at  the 
organizational  unit  level,  are  recorded  by  the  team,  along  with  descriptions  of 
gaps  in  implementation.  When  team  members  have  achieved  their  planned 
coverage  of  data  collection,  the  descriptions  of  gaps  are  validated  with  the 
members  of  the  organization.  This  final  activity  prior  to  rating  allows  team 
members  to  build  confidence  that  their  investigation  has  been  thorough,  and 
the  members  of  the  organization  are  provided  with  an  opportunity  to  correct 
any  perceived  errors  in  the  appraisal  data. 
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2.2.1  Verify  Objective  Evidence 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  appraisal  team  must  establish  a  clear  understanding  of  the  practices 
implemented  in  the  organization.  Typically,  the  organization  provides  a  set  of 
objective  evidence  at  the  beginning  of  the  appraisal  process,  and  the  team  sets 
out  to  verify  the  instances  where  those  practices  are  implemented.  For 
practices  reflecting  project-level  activities,  the  team  must  observe  that  each 
selected  project  in  the  organizational  unit  has  evidence  of  implementation.  For 
practices  reflecting  organization-level  activities,  the  team  must  understand  the 
organization-level  implementation  as  well  as  any  activities  involving  the 
projects  that  indicate  the  implementation  of  the  practice. 


•  Verify  the  appropriateness  of  direct  artifacts  provided  by  each 
instantiation  for  practices  within  the  model  scope  of  the  appraisal. 

•  Verify  the  appropriateness  of  indirect  artifacts  provided  by  each 
instantiation  for  practices  within  the  model  scope  of  the  appraisal. 

•  Verify  the  appropriateness  of  affirmations  provided  by  each  instantiation 
for  practices  within  the  model  scope  of  the  appraisal. 

•  Verify  that  the  implementation  of  each  model  practice  is  supported  by 
direct  artifacts  for  each  instantiation,  and  corroborated  by  indirect  artifacts 
or  affirmations. 

•  Obtain  face-to-face  affirmations  for  (1)  at  least  one  instantiation  for  each 
model  practice  in  the  scope  of  the  appraisal,  or  (2)  at  least  50%  of  the 
practices  corresponding  to  each  specific  and  generic  goal  for  each 
instantiation. 

•  Generate  statements  describing  gaps  in  the  organizational  unit’s  implemented 
practices  relative  to  practices  defined  in  the  reference  model. 


For  practices  implemented  at  the  project  level,  direct  and  indirect  indicators  of 
practice  implementation  must  be  examined  for  every  project  sampled  to 
represent  the  organizational  unit  being  appraised. 

For  practices  implemented  at  the  organization  level,  direct  and  indirect 
indicators  of  practice  implementation  are  examined  in  reference  to  the 
organizational  unit  within  the  scope  of  the  appraisal,  and  not  necessarily  for 
each  project  sampled.  Aspects  of  the  practice  that  are  implemented  at  the 
project  level  must  be  investigated  for  every  project  sampled  to  represent  the 
organizational  unit. 

One  or  more  direct  artifacts  will  be  needed  to  verify  implementation  of  each 
model  practice.  Indirect  indicators  can  include  either  artifacts  or  affirmations. 
A  description  of  these  indicator  types  is  contained  in  activity  1.4.3,  Obtain 
Initial  Objective  Evidence. 

Coverage  criteria  for  face-to-face  affirmations  are  focused  at  the  goal  and 
organizational  unit  level. 
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2.2.1  Verify  Objective  Evidence  (continued) 


Optional  At  the  discretion  of  the  appraisal  team  leader,  verification  of  practices  at  the 

Practices  instantiation  level  may  be  carried  out  solely  by  the  mini-teams.  Team-wide 

review  and  consensus  on  practice  implementation  can  then  focus  on  the 
aggregate-level  characterizations. 

At  the  discretion  of  the  appraisal  team  leader,  the  verification  of  practice 
implementation  at  the  project  level  can  be  reviewed  for  consensus  by  the 
entire  team.  Each  mini-team  provides  an  overview  of  practice  implementation 
indicators  for  each  project  sampled  to  represent  the  organizational  unit. 

A  mix  of  the  two  strategies  above  can  be  used,  selectively  reviewing  targeted 
PAs  in  different  ways,  or  gradually  changing  from  one  strategy  to  the  other  as 
the  team  gains  familiarity  with  the  data  and  the  process. 


Implementation  The  typical  work  products  listed  in  CMMI  models  provide  examples  of 
Guidance  artifacts  that  can  be  used  as  indicators  of  practice  implementation.  However, 

the  model  does  not  distinguish  between  direct  and  indirect  artifacts,  and  these 
are  examples  only  and  are  not  required;  alternatives  can  be  used  for  both 
direct  and  indirect  artifacts. 

Typically,  much  of  the  objective  evidence  required  to  perform  this 
verification  is  provided  in  advance  of  the  on-site  period.  The  primary  focus  of 
data  collection  is  to  permit  the  team  to  verify  that  the  intended  practices  are 
implemented  across  the  organizational  unit.  Where  the  implemented  practices 
differ  from  the  intended  practices,  the  objective  evidence  provided  at  the  start 
of  the  appraisal  process  is  annotated  to  more  accurately  reflect  the 
implemented  process  in  the  organization.  These  annotations  are  typically 
statements  describing  a  gap  in  the  implementation  of  a  model  practice,  some 
of  which  will  eventually  become  findings. 

Where  gaps  exist  in  the  objective  evidence  provided  in  advance,  the  appraisal 
team  is  forced  to  undertake  data  collection  activities  to  populate  the  data  set 
from  scratch.  An  organization  that  has  a  substantial  process  improvement 
infrastructure  in  place  is  expected  to  have  documented  its  implementation  of 
the  model  in  detail.  For  organizations  with  relatively  little  experience  using 
CMMI,  the  cost  of  this  discovery  process  may  be  so  great  that  undertaking  an 
ARC  Class  A  appraisal,  such  as  SCAMPI,  is  not  cost-effective.  For  such 
organizations,  a  Class  B  appraisal  may  be  more  appropriate. 

Only  after  team  members  have  a  clear  understanding  of  the  implemented 
practices  can  they  compare  them  to  the  model  to  characterize  the  extent  to 
which  the  organization  implements  the  practices  in  the  model  or  acceptable 
alternatives.  It  is  expected  that  artifacts  that  result  from  the  performance  of  the 
practice  will  be  available  for  viewing  by  the  team.  These  artifacts,  as  well  as 
face-to-face  interactions  with  members  of  the  organization  enacting  the 
practice,  help  to  verify  that  the  practice  was  enacted  as  the  maintainers  of  the 
organizational  process  intended. 
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2.2.2  Characterize  Implementation  of  Model  Practices 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Once  a  critical  mass  of  evidence  on  practice  implementation  has  been 
verified,  the  team  (or  mini-team)  turns  to  characterizing  the  implementation  of 
model  practices.  For  each  practice  in  the  model  included  in  the  selected  scope, 
and  each  instance  of  expected  use,  the  team  will  document  a  characterization 
of  the  extent  to  which  the  model  practice  (or  an  acceptable  alternative)  has 
been  implemented.  These  project-level  characterizations  are  then  aggregated 
to  the  organizational  unit  level. 

Characterizations  of  practice  implementation  are  used  as  a  means  to  focus 
appraisal  team  effort  on  areas  where  professional  judgment  is  needed,  and  to 
aid  in  reaching  team  consensus  on  the  extent  to  which  practices  are 
implemented. 


•  Characterize,  for  each  instantiation,  the  extent  to  which  reference  model 
practices  are  implemented. 

•  Aggregate  practice  implementation  characterization  values  from  the 
instantiation  level  to  the  organizational  unit  level. 


The  following  table  summarizes  rules  for  characterizing  instantiation-level 
implementations  of  practices.  Consensus  of  at  least  a  subset  of  appraisal  team 
members  (e.g.,  mini-team  members)  is  necessary  for  instantiation-level 
characterizations. 


Label 

Meaning 

Fully  Implemented  (FI) 

•  The  direct  artifact  is  present  and  judged  to  be 
appropriate. 

•  At  least  one  indirect  artifact  and/or  affirmation  exists  to 
confirm  the  implementation. 

•  No  substantial  weaknesses  were  noted. 

Largely  Implemented 
(LI) 

•  The  direct  artifact  is  present  and  judged  to  be 
appropriate. 

•  At  least  one  indirect  artifact  and/or  affirmation  exists  to 
confirm  the  implementation. 

•  One  or  more  weaknesses  were  noted. 

Partially  Implemented 
(PI) 

•The  direct  artifact  is  absent  or  judged  to  be  inadequate. 

•  Artifacts  or  affirmations  suggest  that  some  aspects  of  the 
practice  are  implemented. 

•Weaknesses  have  been  documented. 

Not  Implemented  (NI) 

•  Any  situation  not  covered  above 
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2.2.2  Characterize  Implementation  of  Model  Practices  (continued) 


Parameters 
and  Limits 
(continued) 


Optional 

Practices 


Implementation 

Guidance 


The  following  table  summarizes  rules  for  aggregating  instantiation-level 
characterizations  to  derive  organizational  unit-level  characterizations. 
Consensus  of  all  members  of  the  appraisal  team  is  necessary  for 
organizational  unit-level  characterizations. 

The  column  labeled  “Condition”  is  the  input  condition — the  practice 
implementation  characterizations  for  the  set  of  sampled  projects.  The  column 
labeled  “Outcome”  is  the  resultant  aggregated  practice  implementation 
characterization  at  the  organizational  unit  level. 


Condition 

Outcome 

Remarks 

All  X  (e.g.,  all  LI) 

X 

All  instantiations  have  the  same  characterization. 

All  (LI  or  n) 

LI 

All  instantiations  are  characterized  LI  or  higher. 

Any  PI,  No  NI 

LI  or  PI 

Team  judgment  is  applied  to  choose  LI  or  PI  for 
the  organizational  unit. 

AnyNI 

NI,  PI,  or  LI 

Team  judgment  is  applied  to  choose  NI,  PI,  or  LI 
for  the  organizational  unit. 

While  the  initial  characterization  of  practice  implementation  may  be  proposed 
by  a  mini-team  or  some  subset  of  the  team,  the  following  selections  are 
available; 

•  Instantiation-level  characterization  of  practice  implementation  can  be 
reviewed  by  the  entire  team  for  consensus. 

•  Team-wide  review  and  consensus  on  practice  implementation 
characterization  can  be  reserved  for  the  organizational  unit  level. 

•  A  mix  of  the  two  strategies  above,  tailored  to  match  the  learning  curve  of 
the  team  members  or  to  reflect  the  prioritization  of  particular  PAs,  can  be 
used. 


When  the  team  is  ready  to  perform  the  ratings,  these  characterizations  serve  to 
simplify  the  judgments.  The  team  is  then  able  to  focus  on  the  aggregation  of 
weaknesses  observed  to  determine  the  goal  satisfaction  ratings  (explained  in 
process  2.4).  Situations  where  the  project  has  not  yet  reached  the  appropriate 
point  in  time  where  the  practice  would  be  enacted  are  omitted  from  this 
characterization.  The  appraisal-planning  activities  are  expected  to  prevent 
situations  that  severely  limit  the  examples  of  actual  implementation  for  any 
given  practice. 
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2.2.2  Characterize  Implementation  of  Model  Practices  (continued) 


Implementation 

Guidance 

(continued) 


The  characterization  of  CMMI  practice  implementation  begins  as  soon  as 
sufficient  data  are  available.  It  is  not  necessary  that  data  for  every 
instantiation  be  available  before  the  implementation  of  any  given  practice  can 
be  characterized  at  the  instantiation  level.  However,  before  the 
implementation  of  a  practice  across  the  organizational  unit  can  be 
characterized,  the  instantiation-level  characterizations  must  be  completed. 
Each  instance  of  practice  enactment  is  characterized  using  the  instantiation- 
level  characterization  scheme. 

The  characterization  of  practice  implementation  for  the  organizational  unit  is 
carried  out  using  the  aggregation  rules  summarized  in  the  table  above.  These 
rules  provide  a  basis  for  identifying  the  areas  where  professional  judgment  is 
required,  and  simplify  the  areas  where  the  data  are  unanimous. 
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2.2.3  Validate  Practice  Implementation  Gaps 


Activity 

Description 


Required 

Practices 


Verification  activities  lead  to  statements  summarizing  gaps  (weaknesses)  in 
the  implementation  of  model  practices.  Optionally,  statements  reflecting 
exceptional  implementations  of  model  practices  (strengths)  may  also  be 
generated.  These  statements  can  be  generated  at  various  points  in  the 
appraisal  process,  such  as  when 

•  initial  objective  evidence  is  obtained, 

•  implemented  practices  are  compared  to  the  practices  in  the  reference 
.  model, 

•  the  extent  of  implementation  is  characterized  for  each  project,  or 

•  the  extent  of  implementation  is  characterized  for  the  organizational  unit. 

In  preparation  for  validating  this  information,  the  appraisal  team  generates 
preliminary  findings  that  sununarize  the  practice  implementation  gaps.  The 
preliminary  findings  are  written  in  reference  to  a  single  model  practice,  and 
are  abstracted  to  the  level  of  the  organizational  unit.  The  statements  should 
not  reference  a  specific  individual,  project,  or  other  identifiable  organizational 
sub-unit. 

This  is  still  primarily  a  data  collection  activity,  and  the  intent  is  to  validate  the 
appraisal  team’s  understanding  of  the  processes  implemented  within  the 
organizational  unit.  Feedback  from  participants  may  result  in  modifications  to 
the  appraisal  team’s  inventory  of  objective  evidence.  The  results  of  the 
validation  activity  must  be  considered  in  the  formulation  of  final  findings  and 
goal  ratings.  These  latter  activities  cannot  commence  until  after  the  validation 
activity  has  occurred. 


•  Generate  preliminary  findings  summarizing  gaps  in  practice 
implementation  observed  with  the  organizational  unit  relative  to  reference 
model  practices. 

•  Validate  preliminary  findings  with  members  of  the  organizational  unit. 
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2.2.3  Validate  Practice  Impiementation  Gaps  (continued) 


Parameters 
and  Limits 


Full  appraisal  team  consensus  must  be  reached  on  the  preliminary  findings 
prior  to  providing  them  to  the  organizational  unit  for  validation. 

Preliminary  findings  must  be  corroborated  via  multiple  practice 
implementation  indicator  types  (direct,  indirect,  affirmation).  Areas  where  the 
appraisal  team’s  inventory  of  objective  evidence  is  insufficient  to  satisfy  these 
criteria  may  instead  be  addressed  by  requests  for  additional  information 
needed. 

Preliminary  findings  must  not  refer  to  specific  individuals,  projects,  or 
organizational  sub-units. 

Every  model  practice  characterized  as  either  Not  Implemented  or  Partially 
Implemented,  at  the  organizational  unit  level,  must  have  at  least  one 
preliminary  finding  associated  with  it. 

At  least  one  representative  from  each  project  and  from  any  associated  staff 
functions  must  participate  in  the  set  of  validation  activities. 

Only  appraisal  participants  may  participate  (i.e.,  only  people  who  provided 
data  may  participate  in  validation). 

The  minimum  number  of  validation  sessions  required  is  one,  and  no  more 
than  five  are  recommended,  although  no  maximum  limit  is  specified. 

The  rules  of  confidentiality  and  the  expected  use  of  appraisal  data  must  be 
communicated  to  participants  in  each  validation  activity. 
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2.2.3  Validate  Practice  Implementation  Gaps  (continued) 


Optional  Preliminary  findings  (and  other  appraisal  results)  focused  on  specific  projects, 

Practices  divisions,  or  other  organizational  sub-units  may  be  generated  if  they  are 

reflected  in  the  appraisal  objectives  and  constraints.  This  tailoring  option  also 
requires  that  the  members  of  the  organization  participating  in  the  appraisal  be 
fully  informed  of  the  intended  use  of  the  information  they  provide  to  the 
appraisal  team. 


Implementation  Preliminary  findings  are  the  building  blocks  that  lead  to  the  judgment  of  goal 
Guidance  satisfaction,  and  are  the  detailed  information  that  forms  the  basis  for  the  final 

findings.  As  an  intermediate  artifact  of  the  appraisal  process,  preliminary 
findings  are  used  to  ensure  traceability  between  appraisal  inputs  and  appraisal 
outputs. 

Feedback  from  participants  on  the  preliminary  findings  should  be  solicited  by 
the  appraisal  team  and  considered  for  possible  revisions  to  its  inventory  of 
objective  evidence. 

It  is  not  expected  that  preliminary  findings  will  provide  a  detailed  listing  of 
the  implementation  status  of  every  model  practice  in  every  sampled  project. 
Furthermore,  it  is  not  expected  that  the  preliminary  findings  will  identify  the 
status  of  individual  projects  with  regard  to  practice  implementation  or  goal 
achievement.  An  appraisal  sponsor  may  request  these  more  detailed  appraisal 
results.  The  appraisal  team  leader  should  negotiate  for  the  proper  allocation  of 
time  to  accommodate  this  tailoring  option,  and  the  expectation  that  such 
information  will  be  preserved  at  the  end  of  the  appraisal  should  be  made  clear 
to  all  appraisal  participants. 
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2.2.3  Validate  Practice  Implementation  Gaps  (continued) 


Implementation 

Guidance 

Preliminary 

Findings 

Presentations 


Implementation 

Guidance 

Focus  Groups 


Impiementation 

Guidance 

Survey 

Instrument 


An  interactive  presentation  is  the  most  effective  mechanism  for  validating  the 
preliminary  findings.  The  members  of  the  organization  who  provided  data  to 
the  appraisal  team  are  typically  brought  together  in  a  conference  room,  and  a 
slide  presentation  is  used  to  review  the  preliminary  findings  in  an  effort  to 
invite  people  to  provide  additional  data,  or  express  their  agreement  with  the 
summary  statements.  The  audience  is  often  grouped  by  seniority  in  the 
organization,  and  separate  presentations  are  made  for  practitioners,  project 
managers,  and  middle  managers. 

During  the  presentation,  one  or  more  members  of  the  team  review  the 
preliminary  findings  statements  and  provide  the  audience  with  an  opportunity 
to  comment  or  ask  questions.  The  presenter  uses  only  the  words  crafted  by  the 
appraisal  team  and  avoids  elaborating  on  the  findings  using  his  or  her  own 
words.  When  questions  are  asked  about  a  preliminary  finding,  the  team  leader 
provides  any  clarification  needed  to  understand  what  the  statement  means. 
However,  team  members  avoid  the  appearance  that  they  are  justifying  the 
content  of  the  statement. 

The  detailed  data  that  led  to  the  preliminary  findings  must  be  protected,  and 
there  is  no  negotiation  for  wording  or  eliminating  findings.  The  appraisal 
team  must  record  new  data  made  available  to  them  without  commenting  on 
how  the  data  may  be  interpreted  or  how  the  findings  may  need  to  change. 


As  an  alternative  (or  in  addition)  to  the  presentation,  focus  groups  can  be  used 
to  probe  more  deeply  into  specific  areas  of  the  CMMI  model  with  a  targeted 
audience.  This  would  permit  the  team  to  explore  a  particular  area  in  more 
depth  to  help  sharpen  the  appraisal  results,  or  to  raise  the  visibility  of  the 
results  to  people  who  are  most  informed  on  the  topic.  For  example,  a  focus 
group  conducted  with  project  managers  could  be  an  ideal  environment  to 
validate  (and  gather  more  detailed  data  on)  the  topic  of  project  planning  and 
project  monitoring.  In  contrast,  a  focus  group  composed  of  Engineering 
Process  Group  (EPG)  members  may  be  an  ideal  setting  to  validate  findings 
associated  with  the  organization’s  infrastructure  for  process  improvement. 
The  preliminary  findings  that  relate  to  the  group  may  be  distributed  as 
handouts  or  displayed  using  a  projector,  and  the  participants  can  engage  in  a 
free-form  dialogue  with  the  team  and  amongst  themselves.  Notes  taken  by  the 
members  of  the  team  are  treated  as  any  data  collected  during  an  interview 
would  be. 


Finally,  a  survey  instrument  can  be  used  in  addition  (or  as  an  alternative)  to 
either  of  the  techniques  above.  A  carefully  worded  instrument  that  asks 
respondents  to  rate  their  level  of  agreement  with  the  finding  statement,  and 
provides  an  opportunity  for  written  feedback,  can  provide  a  low-cost  and 
timely  source  of  data  for  the  team. 
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2.3  Document  Objective  Evidence 


Purpose 

Entry  Criteria 

inputs 

Activities 

Outputs 

Outcome 

Exit  Criteria 


Create  lasting  records  of  the  information  gathered  by  identifying  and  then 
consolidating  notes,  transforming  the  data  into  records  that  document  practice 
implementation  as  well  as  strengths  and  weaknesses. 


Planning  activities  for  the  appraisal  are  complete,  including  the  selection  and 
preparation  of  the  appraisal  team.  At  least  one  data  collection  activity  has 
been  conducted,  and  appraisal-relevant  data  are  available  to  record. 


Appraisal  data 

•  notes  taken  during  data  collection  activities  (if  applicable) 

•  annotated  worksheets  or  other  work  aids  containing  data  (if  applicable) 

•  strengths  and  weaknesses  documented  from  previous  activities 

•  data  collection  plan 


2.3.1.  Take/Re  vie  w/Tag  Notes 

2.3.2.  Record  Presence/Absence  of  Objective  Evidence 

2.3.3.  Document  Practice  Implementation  Gaps 

2.3.4.  Review  and  Update  the  Data  Collection  Plan 


•  Updated  appraisal  data 

noted  practice  implementation  gaps  (if  any) 
revised  data  collection  plan  (if  applicable) 
annotated  worksheets 

•  Requests  for  additional  data  (interviewees  or  documents) 


Individual  team  members  understand  the  data  collected  thus  far,  and  have 
information  to  guide  any  needed  subsequent  data  collection. 


All  data  from  the  most  recent  data  collection  session  has  been  captured  as  a 
new  baseline  of  practice  implementation  evidence  or  strength  and  weakness 
statements.  The  data-gathering  plans  have  been  updated  to  reflect  additional 
information  needed  and  topics  that  can  be  removed  from  active  investigation. 


Continued  on  next  page 
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2.3  Document  Objective  Evidence  (continued) 


Key  Points  This  process  has  traditionally  been  the  most  difficult  one  to  manage  during  an 
appraisal.  Members  of  the  team  will  tend  to  vary  a  great  deal  in  their 
productivity  and  style  of  work.  The  team  leader  must  be  very  attentive  to  the 
progress  of  each  team  member,  and  take  effective  corrective  actions  to  ensure 
team  progress. 

Because  of  the  challenging  nature  of  this  activity.  Lead  Appraisers  tend  to 
have  strong  preferences  for  tools  and  techniques  they  have  found  to  be 
successful.  Only  a  high-level  list  of  tools  and  techniques  is  provided  here. 

•  Work  aids  like  wall  charts,  spreadsheet  programs,  and  automated  database 
tools  are  frequently  used  to  help  track  the  status  of  data  collection. 

•  Using  mini-teams,  where  pairs  (or  triplets)  of  team  members  are  assigned 
specific  PAs,  is  a  very  common  practice. 

•  Time  management  is  a  critical  skill  for  this  activity.  Explicitly  reviewing 
the  effort  spent,  in  real  time,  is  a  useful  way  to  focus  the  team. 

•  A  variety  of  techniques  for  structuring  team  notebooks  and  formats  for 
recording  notes  has  been  used. 

•  Team  norms  regarding  techniques  for  managing  debates  and  divergent 
views  are  important,  and  should  be  made  explicit  well  in  advance. 

Metrics  As  mentioned  above,  tracking  the  effort  expended  during  this  activity  (in  real 

time)  is  a  valuable  technique  to  manage  the  team’s  time.  The  ability  to 
quickly  leam  the  rate  at  which  each  team  member  works  is  a  skill  that 
experienced  Lead  Appraisers  develop  using  effort  and  duration  metrics. 

Verification  and  The  method  rules  for  recording  traceability  and  validating  data  provide  a  lot 

Validation  in  the  way  of  verification  and  validation  of  the  appraisal  data.  The  role  of  the 

appraisal  team  leader  in  monitoring  progress  and  the  consensus  decision¬ 
making  process  also  serve  as  important  verification  and  validation  activities. 

Records  All  appraisal  data  are  recorded  with  full  traceability  to  information  sources  as 

well  as  the  model  components  to  which  they  pertain.  The  full  detail  in  this 
traceability  contains  sensitive  information  that  should  not  be  provided  to 
people  outside  of  the  appraisal  team.  The  attribution  of  data  to  individuals  or 
groups  must  never  be  divulged  even  if  some  of  the  detailed  data  are  provided 
to  the  Engineering  Process  Group  at  a  site  for  use  in  process  improvement. 

Tailoring  The  use  of  a  specialized  appraisal  data  management  tool  is  a  common 

tailoring  applied  to  this  activity. 

Continued  on  next  page 
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2.3  Document  Objective  Evidence  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  mechanics  associated  with  the  recording  and  transcription  of  objective 
evidence  are  described  in  this  section.  There  are  many  links  between  these 
mechanics  and  the  data  collection  process,  as  well  as  the  data  verification  and 
validation  process.  It  is  important  to  understand  that  the  data-recording 
process  must  support  these  other  processes,  and  that  the  tools  used  during  an 
appraisal  will  need  to  accommodate  these  linkages.  Typically,  an  integrated 
database  tool  is  used  to  manage  all  appraisal  data  that  results  from  the  analysis 
of  notes  taken  during  data  collection. 


The  most  basic  representation  of  appraisal  data  is  found  in  the  notes  taken  by 
individual  team  members.  These  notes  are  reviewed  and  are  typically 
“tagged”  or  otherwise  processed  before  their  information  content  is 
transformed  into  other  lasting  representations.  The  presence,  absence,  and/or 
appropriateness  of  objective  evidence  is  then  judged  and  recorded  based  on 
the  data  collected.  The  scheme  by  which  this  set  of  records  is  produced  is  an 
important  implementation  choice  made  by  the  appraisal  team  leader,  and  must 
be  well  understood  by  the  team.  Gaps  in  the  implemented  practices  are  also 
recorded,  in  a  consistent  manner  that  ensures  traceability.  Finally,  the  data 
collection  plan  must  be  reviewed  in  light  of  the  changes  in  the  set  of  data 
available  to  the  team,  and  the  remaining  data  needed  to  support  reliable  rating 
judgments. 
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2.3.1  Take/Review/Tag  Notes 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


Optional 

Practices 


As  team  members  examine  data  sources,  they  will  document  what  the 
objective  evidence  is  (referencing  documents,  presentations,  instruments,  and 
interviewee  comments),  as  well  as  why  or  how  the  objective  evidence  meets 
the  intent  of  the  model  practice. 

There  may  be  special  cases  where  team  members  elect  to  record  data  directly 
in  the  objective  evidence  tracking  tool.  In  such  cases  the  team  members  may 
choose  not  to  take  notes  (on  paper  or  in  their  notebooks)  that  describe  the 
objective  evidence. 

For  all  interviews  and  presentations,  the  team  members  must  take  notes  that 
capture  the  objective  evidence  before  they  move  to  the  annotation  of  the 
objective  evidence  tracking  tool. 


•  Record  notes  obtained  from  objective  evidence  data-gathering  sessions. 

•  Relate  notes  to  corresponding  practices  in  the  appraisal  reference  model. 


Every  team  member  present  must  take  notes  during  interviews  and 
presentations.  These  notes  must  cover  all  areas  investigated  during  the 
interview,  and  are  not  limited  to  the  PAs  assigned  to  the  individual  team 
member  (i.e.,  everybody  takes  notes  on  everything). 

During  document  reviews  and  the  review  of  instruments,  notes  must  be  taken 
to  preserve  specific  context  or  focused  references,  if  the  rationale  for 
accepting  the  objective  evidence  is  not  self-evident. 

Whenever  notes  are  taken  in  a  data-gathering  session,  individual  team 
members  should  review  their  notes  immediately  after  the  conclusion  of  the 
session.  The  review  will  focus  on  tagging  significant  items  that  relate  to  one 
or  more  model  practice(s).  This  review  and  tagging  process  must  occur  within 
24  hours  of  the  data-gathering  session. 


Tagging  schemes  (that  show  traceability  to  model  practices)  and  techniques 
for  highlighting  phrases  are  determined  by  the  preferences  of  the  note  taker.  A 
variety  of  formats  for  team  member  notebooks  has  been  devised  to  facilitate 
note  taking  and  tracking  raw  data  during  appraisals.  Frequently,  the  questions 
used  during  an  interview  will  be  printed  and  collated  within  a  team  member 
notebook  that  contains  note-taking  forms  and  other  useful  information  like 
interview  schedules  and  document  lists. 

Notes  can  be  recorded  for  items  that  have  significant  positive  or  negative 
impact  upon  the  enactment  of  processes  within  the  organizational  unit,  even  if 
they  are  not  directly  related  to  model  practices.  These  may  ultimately  be 
reflected  in  non-model  findings  reported  to  the  organizational  unit. 


Continued  on  next  page 
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2.3.1  Take/Review/Tag  Notes  (continued) 


Implementation 

Guidance 


Implementation 

Guidance 

Taking  Notes 


Implementation 

Guidance 

Reviewing 

Notes 


Implementation 

Guidance 

Tagging  Notes 


The  raw  notes  taken  during  an  appraisal  are  treated  as  confidential 
information,  and  may  not  be  provided  to  any  person  outside  of  the  appraisal 
team.  Team  members  are  typically  required  to  destroy  their  notes  in  a  secure 
manner  at  the  conclusion  of  the  appraisal.  This  ensures  that  the  attribution  of 
detailed  information  to  individuals  in  the  organization  cannot  lead  to 
inappropriate  consequences  following  the  appraisal. 


Team  members  actively  take  notes  during  all  data-gathering  sessions.  The 
purpose  is  to  record,  verbatim,  what  the  information  source  reveals  about  the 
implementation  of  practices  in  the  project  or  organization.  Note-taking  is  done 
for  all  types  of  objective  evidence: 

•  The  analysis  of  instruments  yields  information  and  references  regarding 
the  implementation  of  practices,  ideally  with  traceability  to  the  model. 

•  While  reviewing  documents  it  is  often  important  to  note  a  specific  phrase 
or  reference  and  to  record  the  document  name  and  page  number. 

•  When  receiving  presentations,  phrases  or  references  provided  as 
elaboration  on  presentation  material  are  captured  in  notes. 

•  Interviews  are  the  most  intensive  activity  with  regard  to  note  taking.  The 
purpose  is  to  record  what  the  interviewees  said;  not  what  the  team 
member  believes  they  meant. 


The  context  in  which  the  data  are  provided — ^be  it  during  an  interview, 
presentation,  or  in  a  document — ^bears  on  the  proper  interpretation  of  the  facts. 
For  example,  notes  taken  during  an  interview  are  based  on  a  give  and  take 
between  the  interviewer  and  the  interviewee.  The  threads  of  discussion  often 
provide  a  context  that  may  not  be  reflected  in  a  single  excerpt  from  the  middle 
of  the  interview.  Note-takers  should  review  their  work  to  ensure  that  such 
contextual  information  can  be  preserved  at  least  in  their  recollection,  and 
preferably  through  the  annotation  of  the  notes. 


As  notes  are  reviewed,  team  members  often  use  highlighter  pens  or  annotation 
schemes  to  identify  the  most  salient  excerpts.  The  PA  and/or  practice  to  which 
the  information  applies  may  be  written  in  colored  ink  over  the  raw  notes.  All 
notes  should  identify  the  data-gathering  session,  and  the  pages  should  be 
numbered  to  preserve  the  sequence  of  information.  For  notes  taken  during 
interviews,  it  is  often  very  useful  to  draw  a  seating  chart  to  show  where  each 
person  was  sitting  during  the  interview.  Scripts  prepared  in  advance  of 
scheduled  interviews  may  already  be  tagged,  and  can  help  relate  responses  to 
appropriate  sections  of  the  reference  model.  Some  interviewee  responses  may 
deal  with  model  practices  other  than  those  targeted  by  a  given  question,  which 
would  still  necessitate  some  additional  tagging. 
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2.3.2  Record  Presence/Absence  of  Objective  Evidence 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


The  presence  or  absence  of  appropriate  objective  evidence  for  each  model 
practice  in  the  scope  of  the  appraisal  is  determined  based  on  information 
obtained  from  data-gathering  sessions.  Annotations  are  recorded  indicating 
the  source,  relevance,  and  coverage  of  objective  evidence  collected.  In 
situations  where  just  referencing  the  data  source  would  not  make  it  obvious 
why  the  objective  evidence  is  appropriate,  a  comment  can  be  added  to  the 
annotation.  For  example,  when  an  alternative  to  the  typical  work  breakdown 
structure  is  used,  it  may  be  necessary  to  document  why  that  alternative  meets 
the  intent  of  the  model  practice.  Adding  comments  to  the  annotations  can  help 
to  avoid  rehashing  the  rationale  for  accepting  the  objective  evidence  multiple 
times  during  team  discussions. 


Record  the  presence  or  absence  of  appropriate  objective  evidence  collected 
for  each  reference  model  practice. 


The  inventory  of  objective  evidence  (be  it  in  electronic  or  paper  form)  is 
updated  to  reflect  what  the  data  imply  about  the'  implementation  of  particular 
practices.  For  every  practice  within  the  model  scope  of  the  appraisal, 
annotations  indicating  the  presence  or  absence  of  objective  evidence  will  be 
made  throughout  the  appraisal  conduct.  The  annotation  scheme  used  must 
ensure  that  the  record  reveals  the  following  information; 

•  the  project  or  organizational  unit  to  which  the  data  apply 

•  the  specific  or  generic  practice  to  which  the  data  apply 

•  the  type  of  objective  evidence  being  recorded  (i.e.,  direct,  indirect,  or 
affirmation) 

•  whether  the  data  imply  the  presence  or  absence  of  the  objective  evidence 

•  whether  the  data  suggest  that  the  objective  evidence  is  appropriate 

•  comments  about  the  appropriateness  of  the  evidence  (if  needed) 

•  whether  or  not  additional  information  is  needed  before  the  team  can 
characterize  the  extent  to  which  the  practice  is  implemented 

•  a  description  of  what  the  evidence  is,  if  such  a  description  was  not 
provided  by  the  organization  in  advance 
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2.3.2  Record  Presence/Absence  of  Objective  Evidence  (continued) 


Optional  Following  each  verification  session  where  the  presence  or  absence  of 

Practices  objective  evidence  is  recorded,  the  team  reviews  the  judgments  about  each 

new  piece  of  objective  evidence.  This  may  be  useful  in  establishing  a 
common  understanding  of  the  expectations  for  objective  evidence,  especially 
early  in  the  appraisal. 


Implementation  This  activity  represents  the  mechanical  aspects  of  processing  appraisal  data. 
Guidance  and  is  strongly  tied  to  the  activities  described  in  process  2.2,  Verify  and 

Validate  Objective  Evidence.  The  emphasis  of  this  activity  description  is  on 
the  steps  needed  to  update  the  inventory  of  objective  evidence  and  maintain 
traceability  to  data  sources.  The  emphasis  of  the  activity  description  in  Verify 
and  Validate  Objective  Evidence  is  on  the  interpretation  of  data  collected  and 
the  sufficiency  of  objective  evidence  relative  to  the  appraisal  reference  model. 

Team  members  typically  record  the  presence  or  absence  of  appropriate 
objective  evidence  into  tools  such  as  tracking  tables  or  data  consolidation 
worksheets.  Prior  to  the  assignment  of  goal  ratings,  the  entire  team  reviews 
the  status  of  the  objective  evidence  as  reflected  in  the  annotations  made  by 
each  team  member. 

The  data  gathered  during  every  data  collection  session  should  be  related  to  the 
practices  in  use  in  a  project  or  across  the  organization.  In  recording  the 
presence  or  absence  of  objective  evidence,  the  intent  is  to  quickly  inventory 
the  composite  of  factual  information.  Elaboration  about  what  the  data  mean  or 
how  they  relate  to  other  important  issues  is  captured  either  in  notes  or  in  the 
descriptions  of  practice  implementation  gaps  crafted  by  team  members. 
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2.3.3  Document  Practice  Implementation  Gaps 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


The  primary  intent  of  this  activity  is  to  derive,  from  the  objective  evidence 
gathered,  summary  prose  statements  that  describe  the  gap  between  what  the 
objective  evidence  shows  and  what  the  team  was  looking  for  to  support  a 
claim  that  the  model  practice  was  implemented.  The  statements  explain  why 
the  practice  is  not  considered  to  be  Fully  Implemented.  Statements  of  practice 
implementation  gaps  will  be  validated  with  the  organizational  unit  at  a  later 
time. 

Strengths  are  not  recorded  pro  forma  when  practices  are  found  to  be  Fully 
Implemented.  Where  practices  represent  exemplary  implementations  of  the 
model  practices,  the  appraisal  team  will  highlight  these  as  part  of  the  appraisal 
output.  However,  the  primary  focus  of  this  benchmarking  method  is  to  help 
the  organization  verify  the  implementation  of  the  model  and  identify  areas 
where  work  is  needed. 


Describe  in  writing  gaps  in  the  organizational  unit’s  implemented  processes 
relative  to  reference  model  practices. 


For  any  practice  that  is  characterized  as  something  other  than  Fully 
Implemented,  there  must  be  a  prose  statement  explaining  the  gap  between 
what  the  organization  does  and  what  the  model  expects. 

Regardless  of  the  medium  used,  statements  describing  practice 
implementation  gaps  should  be  annotated  with  the  following  identifying 
information: 

•  the  model  component  to  which  the  statement  relates  (i.e.,  PA,  goal,  and 
practice) 

•  the  data  collection  session(s)  in  which  the  information  was  uncovered 

•  the  process  instantiation(s)  to  which  the  statement  applies 

Prose  statements  of  practice  implementation  gaps  presented  to  the 
organizational  unit  in  the  form  of  preliminary  findings  for  validation  must  be 
free  of  references  to  specific  individuals  or  projects. 
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2.3.3  Document  Practice  Implementation  Gaps  (continued) 


Optional  Document  strengths  in  the  implementation  of  model  practices  when  the  team 

Practices  discovers  exemplary  implementations. 

Label  implementation  gaps  as  “opportunities  for  improvement”  to  avoid  the 
potentially  negative  connotations  of  labeling  them  as  weaknesses. 

Document  any  significant  issues  impeding  performance  in  the  organization 
that  do  not  necessarily  map  to  the  CMMI  model.  This  must  be  done 
cautiously,  and  the  number  of  these  issues  should  not  be  larger  than  the 
number  of  model-related  issues  reported  by  the  team. 


Implementation  The  database  used  to  record  the  inventory  of  objective  evidence  may 
Guidance  incorporate  functionality  to  record  practice  implementation  gaps  and 

strengths,  or  a  separate  location  or  tool  may  be  used  if  desired.  Gaps  in 
practice  implementation  should  be  recorded  at  the  level  of  a  particular 
instance  of  a  model  practice.  These  precursors  to  preliminary  findings  are 
more  detailed  and  pointed,  while  all  information  presented  outside  of  the  team 
will  be  aggregated  to  the  goal  and  organizational  unit  level  of  abstraction. 

Strengths  are  only  documented  if  the  implementation  of  a  practice  is 
exceptional,  and  reflects  a  strong  asset  in  the  process  in  use.  An  adequate 
implementation  of  a  model  practice  is  not  necessarily  a  strength.  Team 
members  should  use  their  collective  experience  and  judgment  to  determine 
whether  or  not  they  have  uncovered  an  exemplary  practice  (above  and  beyond 
the  capability  described  in  the  model)  to  highlight  in  the  appraisal  output. 

Gaps  in  practice  implementation  are  documented  if  the  objective  evidence 
indicates  a  missing  component  in  the  process  or  an  inappropriate  practice,  in 
light  of  the  value  the  practice  is  expected  to  add  to  the  achievement  of  the 
goal.  That  is,  practices  that  fail  to  help  the  organization  meet  the  CMMI  goat 
to  which  they  relate  should  have  a  gap  documented  that  explains  why  the  goal 
is  not  met. 
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2.3.4  Review  and  Update  the  Data  Collection  Plan 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


Implementation 

Guidance 


This  activity  is  used  to  continuously  monitor  the  state  of  available  objective 
evidence  and  to  select  the  next  tactic  in  the  pursuit  of  obtaining  full  coverage 
of  the  model  and  the  organizational  unit. 


•  Review  the  inventory  of  objective  evidence  collected  and  the  data 
collection  plan  to  determine  what  additional  objective  evidence  is  still 
needed  for  sufficient  coverage  of  the  model  scope. 

•  Revise  the  data  collection  plan  to  obtain  additional  objective  evidence  for 
instances  where  insufficient  data  are  available  to  judge  the 
implementation  of  reference  model  practices. 

•  Identify  priorities  for  the  upcoming  data  collection  events,  and  reevaluate 
the  feasibility  of  the  schedule  in  light  of  the  current  state  of  the  objective 
evidence. 


This  activity  must  be  enacted  at  least  once  a  day,  and  a  consolidated  summary 
of  the  appraisal  data  collection  status  must  be  available  to  the  team  at  the  start 
of  each  day  during  which  data  collection  events  are  planned. 


In  addition  to  the  daily  status  mentioned  above,  more  frequent  status  checks 
may  be  conducted.  These  interim  status  checks  are  not  aggregated  across  the 
team,  for  a  team-wide  view  of  status,  unless  the  appraisal  team  leader  finds 
that  beneficial. 


The  data  collection  status  summarizes  the  differences  between  the  objective 
evidence  on  hand  and  the  evidence  needed  to  support  the  creation  of  appraisal 
outputs  (e.g.,  ratings).  Annotations  regarding  the  presence  (and 
appropriateness)  of  objective  evidence  allow  the  team  to  inventory  the  state  of 
the  “knowledge  base.”  This  status  then  drives  requirements  for  the  collection 
of  more  data,  which  must  be  met  by  the  data  collection  plan.  The  annotation 
of  the  inventory  of  objective  evidence  is  described  in  process  2.2,  Verify  and 
Validate  Objective  Evidence. 

The  plan  for  future  data  collection  should  be  revisited  and  updated  as 
necessary.  There  may  be  several  situations  in  which  additional  data  are 
required  for  the  team  to  sufficiently  characterize  the  implementation  of 
reference  model  practices.  The  following  are  examples  of  such  situations: 

•  The  process  of  reconciling  new  data  with  old  may  identify  conflicts  or 
ambiguities  in  the  data  that  require  clarification. 

•  The  search  for  objective  evidence  may  lead  to  the  discovery  of  one  or 
more  previously  undocumented  practice(s)  in  the  organization. 

•  Attempts  to  confirm  the  use  of  a  particular  practice  or  tool  in  a  project 
may  have  been  unsuccessful. 


Continued  on  next  page 
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2.3.4  Review  and  Update  the  Data  Collection  Plan  (continued) 


Implementation 

Guidance 

(continued) 


Prioritizing  data  needs  and  allocating  data  collection  effort  to  particular  data 
collection  events  are  ongoing  activities  that  the  appraisal  team  leader  is 
responsible  for  overseeing.  The  data  collection  status  summary  may  be 
performed  by  the  appraisal  team  leader  and  reported  to  the  team  members,  or 
the  appraisal  team  leader  may  elect  to  have  each  mini-team  perform  this 
activity  for  the  PAs  it  is  assigned. 

Specific  information  needed  to  resolve  ambiguities  or  conflicts  in  the  existing 
data  should  be  documented  for  follow-up  by  one  or  more  members  of  the 
team.  For  detailed  data  items  that  have  a  limited  scope  of  impact,  the  notes  of 
individual  team  members  may  be  adequate  to  document  the  data  needed.  For 
example,  whether  or  not  a  particular  person  is  involved  in  a  meeting,  or 
reviews  a  given  document,  can  be  confirmed  by  a  simple  question  asked 
during  an  on-call  interview.  Therefore,  a  note  made  by  an  individual  team 
member  to  make  sure  the  question  is  asked  nniay  suffice.  In  contrast,  if 
conflicting  information  is  uncovered  about  whether  or  not  a  given  event 
occurred,  like  a  meeting,  more  visibility  of  this  conflict  may  be  needed  among 
the  team  members  to  understand  why  the  information  collected  thus  far  is  not 
internally  consistent.  In  such  a  case,  the  person(s)  responsible  for  the  PA 
where  that  practice  resides  may  need  to  alert  the  team  to  the  conflicting  data 
and  facilitate  a  team  discussion  to  seek  clarity,  as  well  as  additional  data.  This 
may  lead  to  the  crafting  of  a  specific  interview  question,  which  is  used  in  a 
standard  interview. 

The  data  collection  plan  and  inventory  of  objective  evidence  provide  a  means 
for  the  appraisal  team  to  continuously  monitor  progress  toward  sufficient 
coverage  of  reference  model  practices  in  preparation  for  rating.  Estimates  of 
the  additional  data  collection  effort  should  be  regularly  reviewed.  If  the 
feasibility  of  the  appraisal  schedule  is  called  into  question,  a  replanning  effort 
may  be  necessary  (as  described  in  activity  1.5.3,  Replan  Data  Collection). 
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2.4  Generate  Appraisal  Results 


Purpose 

Rate  goal  satisfaction  based  upon  the  extent  of  practice  implementation 
throughout  the  organizational  unit.  The  extent  of  practice  implementation  is 
judged  based  on  validated  data  (e.g.,  direct,  indirect,  and  affirmation  objective 
evidence)  collected  from  the  entire  representative  sample  of  the  organizational 
unit.  The  rating  of  capability  levels  and/or  maturity  levels  is  driven  by  the 
goal  satisfaction  ratings. 

Entry  Criteria 

The  set  of  validated  preliminary  findings,  statements  of  practice 
implementation  gaps,  and/or  tabulations  of  validated  objective  evidence  of 
practice  implementation  on  which  they  are  based  are  available.  Team 
members  are  confident  that  they  have  obtained  all  the  pertinent  data  they  will 
need  to  make  rating  judgments.  The  data  obtained  completely  covers  the 
practices  within  the  defined  CMMI  model  scope  and  the  entire  representative 
sample  selected  for  the  organizational  unit. 

Inputs 

Appraisal  data 

•  validated  preliminary  findings 

•  tabulations  of  objective  evidence  of  practice  implementation 

•  annotated  worksheets,  checklists,  working  notes 

Activities 

2.4. 1  Derive  Findings  and  Rate  Goals 

2.4.2a  Determine  Process  Area  Capability  Level 

2.4.3a  Determine  Capability  Profile 

2.4.2b  Determine  Satisfaction  of  Process  Areas 

2.4.3b  Determine  Maturity  Level 

2.4.4  Document  Appraisal  Results 

Outputs 

•  Final  findings 

•  Recorded  rating  decisions 

Outcome 

A  formal  rating  decision  for  each  reference  model  component  that  was 
planned  to  be  rated,  and  for  which  the  team  obtained  complete  or  sufficient 
data 

Exit  Criteria 

Ratings  against  all  components  per  the  plan  have  been  made  and  recorded. 

Continued  on  next  page 
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2. 


Generate  Appraisal  Results  (continued) 


Key  Points  The  judgment  of  goal  satisfaction  is  based  upon  and  traceable  to  the  extent  of 
the  in^lementation  of  practices  associated  with  that  goal  (or  alternative 
practices  contributing  equivalently  to  goal  satisfaction). 

Success  in  this  activity  is  driven  by  team  members’  ability  to  limit  their  focus 
to  the  data  that  support  the  judgments,  and  to  avoid  issues  that  threaten  their 
ability  to  be  objective.  This  activity  can  create  a  great  deal  of  stress  for  team 
members  under  pressure  to  help  their  organization  “do  well”;  the  team  leader 
must  skillfully  facilitate  this  activity  when  external  pressures  exist. 

Tools  and  There  is  a  significant  amount  of  data  to  review  in  making  each  round  of 

Techniques  judgments.  Rating  worksheets  and  automated  support  tools  facilitate  the 

team’s  decision-making  process  by  presenting  necessary  data  in  a  concise, 
well-organized  manner.  When  controversial  issues  are  encountered,  the  team 
leader  must  actively  facilitate  to  ensure  that  the  team  remains  focused  on  the 
pertinent  issues.  Strategic  rest  breaks,  and  sequencing  and  pacing  critical 
discussions,  are  often  keys  to  success. 

Metrics  •  Planned  versus  actual  effort  for  each  component  rated 

•  Number  of  model  components  rated  satisfied  or  unsatisfied 

Verification  and  The  team  leader  verifies  that  the  rating  process  was  performed  in  accordance 

Validation  with  the  method  rules  and  the  rating  baseline  selected  and  documented  in  the 

appraisal  plan.  Work  aids  used  to  record  the  team  judgments  help  ensure 
traceability  to  the  basis  for  the  rating  judgments. 

Records  A  worksheet  or  other  work  aid  may  be  used  to  make  a  record  of  the  rating 

decisions.  A  Process  Area  Profile  is  often  an  effective  means  of  recording  and 
communicating  these  results. 

Tailoring  The  method  provides  tailoring  options  for  rating  additional  model 

components.  The  minimum  requirement  is  to  rate  the  specific  and  generic 
goals  associated  with  each  PA  in  the  scope  of  the  appraisal.  In  addition,  the 
sponsor  may  request  that  maturity  level  and/or  capability  level  ratings  be 
performed  and  reported.  Through  negotiation  between  the  appraisal  team 
leader  and  the  appraisal  sponsor,  a  decision  to  rate  individual  practices  can 
also  be  made. 

Continued  on  next  page 
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2.4  Generate  Appraisal  Results  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  rating  judgments  made  by  the  appraisal  team  members  are  dependent  on 
the  quality  of  the  data  available  to  them,  as  well  as  their  ability  to  reliably 
judge  the  implementation  and  institutionalization  of  practices  in  the 
organization  that  relate  to  the  CMMI  model.  All  the  processes  previously 
described  contribute  to  the  team’s  ability  to  effectively  execute  this  process. 
The  Analyze  Requirements  process  establishes  the  rating  baseline,  the 
organizational  unit  to  which  ratings  will  apply,  and  the  purpose  for  which  the 
ratings  will  be  used.  The  Develop  Appraisal  Plan  process,  in  conjunction  with 
the  Obtain  and  Analyze  Initial  Objective  Evidence  and  Prepare  for  Collection 
of  Objective  Evidence  processes,  determine  the  sample  of  the  organizational 
unit  for  which  data  will  be  collected  and  from  which  the  ratings  will  be 
determined.  The  Select  and  Prepare  Team  process  ensures  that  the  team  has 
sufficient  knowledge  and  skills  to  interpret  the  data  and  arrive  at  sound  rating 
judgments.  The  Examine  Objective  Evidence  and  Document  Objective 
Evidence  processes  provide  the  basic  information  that  is  needed  to  support 
judgments  in  a  form  that  facilitates  making  the  judgments.  The  Verify  and 
Validate  Objective  Evidence  process  characterizes  the  extent  to  which  the 
organization  implements  practices  in  the  model  (or  acceptable  alternatives) 
and  validates  findings  describing  any  weaknesses  associated  with  the  practice 
implementations.  Upon  the  successful  execution  of  these  processes,  the  team 
is  ready  to  rate  the  satisfaction  of  goals  dependent  on  those  practices. 


The  required  and  fundamental  rating  activity  involves  making  team  judgments 
about  goal  satisfaction  for  each  and  every  specific  and  generic  goal  within  the 
model  scope  defined  in  the  rating  baseline.  Once  goal  satisfaction  has  been 
determined,  optional  rating  activities  can  be  performed  in  accordance  with  the 
defined  rating  baseline  and  the  selected  model  representation(s)  (continuous, 
staged,  or  both).  The  first  optional  activity  focuses  on  rolling  up  goal 
satisfaction  to  PA  ratings.  The  team  determines  a  PA  capability  level  rating  (0 
through  5)  for  each  PA  in  the  continuous  representation  that  is  within  the 
appraisal  scope,  and/or  the  team  determines  a  Satisfied/Unsatisfied  rating  for 
each  PA  in  the  staged  representation  that  is  within  the  appraisal  scope.  The 
second  optional  activity  continues  the  rating  roll  up  to  cover  all  PAs  within 
the  selected  model  scope.  In  the  case  of  the  continuous  representation  the 
team  creates  a  profile  showing  the  capability  levels  for  all  PAs  considered. 
The  profile  can  then  be  used  to  compute  a  maturity  level  through  the 
equivalent  staging  described  in  the  model.  In  the  case  of  the  staged 
representation  the  team  assigns  a  maturity  level  rating  (1  through  5) 
corresponding  to  the  highest  level  in  the  model  for  which  all  applicable  PAs 
have  been  rated  as  satisfied.  The  optional  activities  described  in  2.4.2a  and 
2.4.3a  cover  the  continuous  representation;  those  in  2.4.2b  and  2.4.3b  cover 
the  staged  representation.  As  indicated,  these  options  are  not  mutually 
exclusive. 
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2.4.1  Derive  Findings  and  Rate  Goais 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  judgments  made  about  goal  satisfaction  are  driven  by  the  findings  that 
were  documented  by  the  appraisal  team  and  validated  by  appraisal 
participants.  The  preliminary  findings  focus  on  gaps  in  the  implementation  of 
practices.  When  performing  goal  ratings,  the  team  must  judge  whether  or  not 
these  gaps  in  the  implementation  of  practices  (in  aggregate)  threaten  the 
organization’s  ability  to  satisfy  the  goals  associated  with  the  practices. 


•  Derive  final  findings  using  preliminary  findings  statements,  feedback 
from  validation  activities,  and  any  additional  objective  evidence  collected 
as  a  result  of  the  validation  activity. 

•  Rate  each  specific  goal  and  generic  goal  within  the  reference  model  scope 
of  the  appraisal,  based  on  the  practice  implementation  characterizations  at 
the  organizational  unit  level,  as  well  as  the  aggregation  of  weaknesses 
associated  with  that  goal. 

•  Obtain  appraisal  team  consensus  on  the  practice  implementation 
characterizations,  findings  statements,  and  ratings  generated  for  the 
organizational  unit  level. 


When  deriving  final  findings,  the  aim  is  to  create  goal-level  statements  that 
summarize  the  gaps  in  practice  implementation.  These  statements  must  be 
abstracted  to  the  level  of  the  organizational  unit,  and  cannot  focus  on 
individual  projects  (unless  the  tailoring  option  for  project-specific  findings 
has  been  agreed  upon  during  planning). 

If  there  are  no  findings  that  document  the  weaknesses  associated  with  a  goal, 
the  goal  must  be  satisfied. 

The  goal  is  rated  Satisfied  if 

•  all  associated  practices  are  characterized  at  the  organizational  unit  level  as 
either  Largely  Implemented  or  Fully  Implemented,  and 

•  the  aggregation  of  weaknesses  associated  with  the  goal  does  not  have  a 
significant  negative  impact  on  goal  achievement. 

For  a  goal  to  be  rated  as  Unsatisfied,  the  team  must  be  able  to  describe  how 
the  set  of  weaknesses  (or  single  weakness)  led  to  this  rating. 


Continued  on  next  page 
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2.4.1  Derive  Findings  and  Rate  Goals  (continued) 


Optional  Findings  statements  and  satisfaction  ratings  may  be  specified  at  the  level  of 

Practices  individual  practices  if  the  appraisal  sponsor  specifically  requests  this  tailoring 

option.  These  practice-level  ratings  must  be  based  on  the  extent  to  which  the 
implemented  practice  (or  the  absence  of  implementation)  supports  the 
achievement  of  the  related  goal.  The  use  of  informative  material  to  form  a 
checklist  is  explicitly  discouraged.  A  rating  algorithm  for  practices  that  does 
not  have  a  demonstrable  link  to  PA  goals  would  depart  from  the  intended  use 
of  CMMI  components. 


Implementation  Any  endeavor  that  results  in  producing  a  score,  grade,  or  rating  is  by 
Guidance  definition  an  area  of  sensitivity  to  those  affected  by  the  outcome.  An  objective 

and  clear-cut  basis  for  assigning  a  rating  lessens  this  sensitivity  and  results  in 
a  more  consistent  basis  of  comparison  among  the  organizational  units  and 
goals  rated.  Judgments  made  prior  to  and  during  the  rating  process  should  be 
based  on  observable  facts  and  should  be  made  at  the  lowest  level  of 
abstraction  that  makes  sense.  In  the  case  of  CMMI,  the  lowest  level  of 
abstraction  is  characterizing  the  extent  of  practice  implementation  for  each 
process  instantiation  within  the  representative  sample.  Characterizations  made 
at  the  instantiation  level  are  aggregated  into  a  characterization  of  the  extent  of 
practice  implementation  throughout  the  organization,  as  described  earlier  in 
process  2.2,  Verify  and  Validate  Objective  Evidence.  The  judgment  of  goal 
satisfaction  is  then  based  upon,  and  directly  traceable  to,  the  extent  of 
implementation  of  practices  associated  with  that  goal  (or  alternative  practices 
contributing  equivalently  to  goal  satisfaction). 

Findings  should  be  phrased  in  terms  that  best  facilitate  decision  making  by  the 
appraisal  sponsor  and  taking  action  upon  the  appraisal  results. 
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2.4.2a  Determine  Process  Area  Capability  Level 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


When  using  the  continuous  representation  of  a  CMMI  model,  the  team  may 
make  rating  judgments  about  each  PA  (and  associated  capability  level)  within 
the  scope  of  the  appraisal.  Assigning  capability  level  ratings  is  an  optional 
activity,  selected  at  the  discretion  of  the  appraisal  sponsor  and  documented  in 
the  appraisal  input. 


Rate  the  capability  levels  for  each  PA  within  the  scope  of  the  appraisal,  based 
upon  the  highest  level  and  all  levels  below  for  which  its  specific  goals  and  the 
generic  goals  within  the  appraisal  scope  have  been  satisfied  (if  this  rating 
option  was  selected  during  planning). 


The  table  below  provides  the  criteria  for  deriving  the  capability  level  rating 
for  each  PA. 


Capability 

Level 

Engineering  Process  Areas 

Other  Process  Areas 

0 

Default  Rating 

Default  Rating 

1 

Generic  goal  for  capability  level  1  is 
rated  Satisfied. 

All  specific  goals  are  rated  Satisfied  - 
including  base  practices  only. 

Generic  goal  for  capability 
level  1  is  rated  Satisfied. 

All  specific  goals  are  rated 
Satisfied. 

2 

Generic  goals  for  capability  levels  1 
and  2  are  rated  Satisfied. 

All  specific  goals  are  rated  Satisfied  - 
including  specific  practices  at 
capability  levels  1  and  2. 

Generic  goals  for  capability 
levels  1  and  2  are  rated 
Satisfied. 

All  specific  goals  are  rated 
Satisfied. 

3 

Generic  goals  for  capability  levels  1, 2, 
and  3  are  rated  Satisfied. 

All  specific  goals  are  rated  Satisfied  - 
including  specific  practices  at 
capability  levels  1, 2,  and  3. 

Generic  goals  for  capability 
levels  1,  2,  and  3  are  rated 
Satisfied. 

All  specific  goals  are  rated 
Satisfied. 

4 

Generic  goals  for  capability  levels  1, 2, 
3,  and  4  are  rated  Satisfied. 

All  specific  goals  are  rated  Satisfied  - 
including  specific  practices  at 
capability  levels  1, 2,  and  3. 

Generic  goals  for  capability 
levels  1,  2,  3,  and  4  are  rated 
Satisfied. 

All  specific  goals  are  rated 
Satisfied. 

5 

Generic  goals  for  capability  levels  1, 2, 
3,  4,  and  5  are  rated  Satisfied. 

All  specific  goals  are  rated  Satisfied  - 
including  specific  practices  at 
capability  levels  1, 2,  and  3. 

Generic  goals  for  capability 
levels  1,  2,  3, 4,  and  5  are 
rated  Satisfied. 

All  specific  goals  are  rated 
Satisfied. 

Continued  on  next  page 


Page  11-116 


CMU/SEi-2001-HB-001 


2.4.2a  Determine  Process  Area  Capability  Level  (continued) 


Optional  The  rating  of  PA  capability  levels  may  be  carried  out  top  down  or  bottom  up, 

Practices  as  described  below. 

The  bottom  up  approach  uses  the  following  sequence; 

•  Judge  whether  or  not  the  PA  can  be  considered  to  be  at  capability  level  1, 
based  on  the  satisfaction  of  specific  and  generic  goals.  In  this  case,  only 
the  base  practices  would  be  considered  in  rating  goals. 

•  Judge  whether  or  not  the  PA  can  be  considered  to  be  at  capability  level  2, 
based  on  the  satisfaction  of  specific  and  generic  goals.  In  this  case,  the 
advanced  practices  for  capability  level  2  must  be  considered  in  rating  the 
goals  of  the  Engineering  PAs. 

•  Proceed  incrementally  until  the  team  reaches  a  point  at  which  the  goals 
cannot  be  rated  as  satisfied. 

The  top  down  approach  uses  the  following  sequence: 

•  Begin  at  the  highest  desired  capability  level  (which  was  determined 
during  appraisal  planning)  and  judge  whether  or  not  the  PA  can  be 
considered  to  be  operating  at  that  capability  level. 

•  If  the  PA  is  not  at  the  highest  desired  capability  level,  consider  whether  or 
not  it  can  be  judged  to  be  operating  at  the  next  lower  level. 

•  Proceed  incrementally  until  the  team  reaches  a  point  at  which  all  of  the 
relevant  goals  are  rated  as  satisfied,  or  goal  ratings  lead  to  capability  level 
0. 


Implementation  The  presence  of  advanced  practices  in  the  Engineering  PAs  creates  a  nuance 
Guidance  in  the  rating  process  that  can  be  complicated  for  some  appraisal  team 

members.  If  team  members  have  only  worked  with  the  staged  representation 
in  the  past,  it  is  important  that  the  appraisal  team  leader  covers  this  nuance 
during  team  training,  and  prevents  confusion  during  the  rating  process. 

Goal  satisfaction  is  a  judgment  based  on  the  implementation  of  practices  that 
map  to  the  goal.  In  rating  the  satisfaction  of  specific  goals  in  the  Engineering 
PAs,  the  set  of  specific  practices  that  relates  to  the  goals  differs  for  capability 
levels  0,  1,  2,  and  3  through  5.  That  is,  depending  on  the  capability  level  at 
which  the  rating  is  performed,  there  are  up  to  4  unique  sets  of  specific 
practices  associated  with  these  specific  goals  that  must  be  considered. 

The  appraisal  team  leader  is  responsible  for  selecting  one  of  the  two  optional 
rating  approaches  described  above,  and  should  facilitate  this  session  carefully 
to  prevent  confusion  among  team  members. 
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2.4.2b  Determine  Satisfaction  of  Process  Areas 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


Implementation 

Guidance 


When  using  the  staged  representation  of  a  CMMI  model,  the  team  may  derive 
the  satisfaction  of  PAs  from  the  set  of  goal  satisfaction  judgments.  Assigning 
PA  satisfaction  ratings  is  an  optional  activity,  selected  at  the  discretion  of  the 
appraisal  sponsor  and  documented  in  the  appraisal  input. 


Rate  the  satisfaction  of  each  PA  in  the  scope  of  the  appraisal  based  on  the 
ratings  of  the  goals  within  each  PA,  if  this  rating  option  was  selected  during 
planning. 


PAs  may  be  assigned  rating  values  of  Satisfied,  Unsatisfied,  Not  Applicable, 
or  Not  Rated. 

A  PA  is  rated  Satisfied  if  and  only  if  all  of  its  specific  goals  and  generic  goals 
are  rated  Satisfied. 

If  even  one  of  the  goals  in  a  PA  is  rated  Unsatisfied,  then  the  PA  is  rated 
Unsatisfied. 

When  a  PA  is  determined  to  be  outside  of  the  organizational  unit’s  scope  of 
work,  the  PA  is  designated  as  Not  Applicable  and  is  not  rated.  The 
identification  of  a  PA  as  Not  Applicable  must  occur  during  the  planning  of  the 
appraisal. 

When  a  PA  is  outside  of  the  appraisal  scope,  or  if  the  associated  set  of 
objective  evidence  does  not  meet  the  defined  criteria  for  sufficient  data 
coverage,  the  PA  is  designated  as  Not  Rated  and  is  not  rated.  The  criteria  for 
sufficient  data  coverage  are  described  in  activity  2.2.1,  Verify  Objective 
Evidence. 


A  profile  to  summarize  the  satisfaction  of  goals  may  be  created  to  provide 
further  insight  about  the  rating  outcomes.  Where  a  PA  is  rated  as  Unsatisfied, 
this  more  detailed  view  of  the  rating  outcomes  may  provide  focus  and 
visibility  at  a  lower  level  of  detail. 


PA  satisfaction  is  a  direct  function  of  goal  satisfaction.  A  PA  is  rated  as 
Satisfied  if  every  goal  contained  in  the  PA  is  rated  as  Satisfied.  A  PA  is  rated 
as  Unsatisfied  if  any  goal  is  rated  as  Unsatisfied.  This  ensures  that  one  or 
more  weaknesses  exist  that  serve  to  explain  why  the  goal  and  therefore  the  PA 
are  not  satisfied. 

PA  ratings  need  not  be  reported  to  appraisal  participants,  if  the  sponsor  does 
not  wish  to  disclose  these  results.  However,  a  documented  output  from  this 
rating  activity,  if  it  is  performed,  is  a  required  component  in  the  Appraisal 
Record. 
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2.4.3a  Determine  Capability  Profile 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


Implementation 

Guidance 


When  using  the  continuous  representation  of  a  CMMI  model,  the  team  may 
determine  a  Capability  Profile  that  graphically  depicts  the  capability  level 
ratings  assigned  to  each  PA  within  the  scope  of  the  appraisal.  The  generation 
of  a  Capability  Profile  is  an  optional  activity,  selected  at  the  discretion  of  the 
appraisal  sponsor  and  documented  in  the  appraisal  input. 

Generate  a  Capability  Profile  depicting  the  capability  level  attained  for  each 
PA  within  the  scope  of  the  appraisal,  if  this  rating  option  was  selected  during 
planning. 


A  simple  bar  chart  is  used  for  this  display.  Each  PA  is  represented  in  a  single 
bar  along  the  horizontal  axis,  and  the  vertical  axis  represents  the  capability 
level  dimension.  The  height  of  each  bar  communicates  the  capability  level  of 
the  PA  represented. 

Capability  levels  take  only  the  values  0,  1,  2,  3,  4,  or  5.  Intermediate  values 
(e.g.,  2.7)  are  not  defined  for  this  appraisal  outcome,  and  any  embellishment 
of  the  Capability  Profile  with  such  values  is  outside  the  boundaries  of 
SCAMPI. 


A  profile  to  summarize  the  satisfaction  of  goals  may  be  created  to  provide 
further  insight  about  the  rating  outcomes.  In  situations  where  a  PA  capability 
level  rating  does  not  reflect  the  desired  outcome,  this  more  detailed  view  may 
provide  focus  and  visibility  at  a  lower  level  of  detail. 

CMMI  provides  for  equivalent  staging,  whereby  a  Capability  Profile  can  be 
used  to  derive  an  equivalent  maturity  level  rating  (see  activity  2.4.3b, 
Determine  Maturity  Level). 


A  presentation  template  referred  to  as  a  Capability  Profile  is  typically  used  to 
communicate  the  aggregate  level  rating  results  to  the  sponsor  and  others 
designated  by  the  sponsor. 

Comparing  different  PAs  with  respect  to  their  relative  capability  level  ratings 
may  be  informative  in  discussing  trends  or  patterns  in  the  organization. 

This  activity  may  be  omitted  entirely,  as  it  is  a  tailoring  option.  If  a  Capability 
Profile  is  to  be  derived,  the  ratings  reflected  in  the  profile  are  derived  as 
described  in  activity  2.4.2b,  Determine  Process  Area  Capability  Level. 
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2.4.3b  Determine  Maturity  Level 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


Optional 

Practices 

Implementation 

Guidance 


Historically,  one  of  the  most  visible  outcomes  of  an  appraisal  has  been  the 
maturity  level  rating  assigned.  The  determination  of  a  maturity  level  rating  is 
straightforward,  and  is  derived  mechanically  ffom  the  ratings  assigned  at  the 
lower  levels  of  detail.  Assigning  a  maturity  level  rating  is  an  optional  activity, 
selected  at  the  discretion  of  the  appraisal  sponsor  and  documented  in  the 
appraisal  input. 


Rate  the  maturity  level  based  on  the  ratings  assigned  to  PAs,  if  this  rating 
option  was  selected  during  planning. 


The  maturity  level  determined  is  the  highest  level  at  which  all  PAs  contained 
within  the  maturity  level,  and  within  all  lower  maturity  levels,  are  rated  as 
Satisfied  or  Not  Applicable.  The  single  exception  to  this  rule  is  that  the 
maturity  level  3  goal  for  each  applicable  maturity  level  2  PA  must  also  be 
rated  Satisfied  for  a  maturity  level  rating  of  3  or  higher  to  be  determined. 

When  using  continuous  representations,  CMMI  provides  for  equivalent 
staging,  whereby  a  Capability  Profile  can  be  used  to  derive  an  equivalent 
maturity  level  rating.  A  maturity  level  for  a  continuous  representation  is 
achieved  if  the  Capability  Profile  is  at  or  above  the  target  profile  for  all  PAs 
for  that  maturity  level  and  all  lower  maturity  levels  in  the  equivalent  staging, 
excepting  those  PAs  that  are  designated  as  Not  Applicable.  The  equivalence 
of  particular  Capability  Profiles  and  particular  maturity  levels  is  addressed  in 
an  appendix  to  the  CMMI  model. 

To  determine  a  maturity  level  as  an  output  of  the  appraisal,  the  model  scope 
of  the  appraisal  must  include  the  minimum  set  of  PAs  required  by  the  CMMI 
model.  Please  refer  to  the  tailoring  section  of  the  CMMI  model  for  guidelines 
on  what  the  minimally  acceptable  scope  of  the  model  is  for  each  maturity 
level. 


None. 


This  activity  may  be  omitted  entirely,  as  it  is  a  tailoring  option.  If  a  maturity 
level  is  to  be  reported,  the  PA  ratings  that  form  the  basis  for  the  maturity  level 
rating  are  derived  as  described  in  activity  2.4.2b,  Determine  Satisfaction  of 
Process  Areas. 
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2.4.4  Document  Appraisal  Results 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 

Implementation 

Guidance 


The  results  of  the  appraisal  conduct  must  be  documented  for  reporting.  Verbal 
reports  of  the  rating  outcomes  or  face-to-face  explanations  of  implementation 
gaps  discovered  by  the  team  are  not  sufficient  to  communicate  appraisal 
results. 


•  Document  the  final  findings. 

•  Document  the  rating  outcome(s). 

•  Document  the  Appraisal  Disclosure  Statement  (ADS). 


The  ADS  and  the  set  of  appraisal  outputs  agreed  upon  with  the  appraisal 
sponsor  must  be  documented.  These  appraisal  outputs  may  exclude  all  ratings, 
and  the  sponsor  is  free  to  select  and  disclose  a  variety  of  appraisal  outcomes, 
as  specified  in  the  activities  of  this  process. 

Regardless  of  the  needs  of  the  sponsor,  the  ADS,  the  goal  ratings,  and  the 
associated  findings  must  be  documented  as  a  part  of  the  appraisal  information 
returned  to  the  CMMI  Steward. 


Any  optional  outputs  requested  by  the  appraisal  sponsor  are  also  created 
during  this  activity. 


This  activity  is  focused  on  collecting  and  documenting  the  results  of  prior 
activities  related  to  the  generation  of  findings  and  ratings.  Depending  on  the 
planned  recipients  of  the  results,  multiple  forms  of  the  results  may  be  needed. 
Certain  data  may  not  be  appropriate  for  all  audiences,  or  the  style  or  language 
of  the  results  may  need  to  be  adjusted  to  best  fit  the  needs  of  the  recipients. 

The  documented  appraisal  results  are  typically  provided  in  a  final  findings 
presentation,  described  in  activity  3.1.1,  Present  Final  Findings. 
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3.1  Deliver  Appraisal  Results 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Provide  credible  appraisal  results  that  can  be  used  to  guide  actions.  Represent 
the  strengths  and  weaknesses  of  the  processes  in  use  at  the  time.  Provide 
ratings  (if  planned  for)  that  accurately  reflect  the  capability  level  or  maturity 
level  of  the  processes  in  use. 


•  Objective  evidence  has  been  validated  (through  the  team  process). 

•  Preliminary  findings  have  been  validated. 

•  Ratings  have  been  determined  (for  model  components  selected  for  rating). 

•  Final  findings  have  been  created  and  reviewed  by  the  team. 


•  Appraisal  data 

final  findings 
ratings 

•  Appraisal  artifacts 
-  appraisal  input 

appraisal  plan 


3.1.1  Present  Final  Findings 

3.1.2  Conduct  Executive  Session(s) 

3.1.3  Plan  for  Next  Steps 


•  Documented  final  findings 

•  Final  report  (if  requested) 

•  Recommendations  report  (if  requested) 


•  The  sponsor  and  the  appraised  organizational  unit  are  provided  with  the 
results  of  the  appraisal. 

•  A  valid  and  reliable  characterization  of  the  current  state  of  the  processes 
in  use  across  the  organizational  imit  is  documented. 


•  Appraisal  results  are  delivered  to  the  appraisal  sponsor  and  organizational 
unit. 

•  An  executive  session  is  conducted,  if  appropriate. 


Continued  on  next  page 
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3.1  Deliver  Appraisal  Results  (continued) 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification  and 
Validation 


Records 


Tailoring 


The  appraisal  results  are  intended  to  support  decision  making,  and  need  to  be 
delivered  in  a  way  that  promotes  appropriate  actions.  Whether  the  appraisal 
was  conducted  for  internal  process  improvement,  supplier  selection,  or 
process  monitoring  purposes,  the  delivery  of  results  should  facilitate  the 
actions  that  will  be  driven  by  the  information. 


Templates  containing  standard  information  for  use  in  a  final  findings  briefing 
are  provided  to  all  SCAMPI  Lead  Appraisers.  Experienced  appraisers 
frequently  use  electronic  (database)  tools  that  support  the  transformation  of 
raw  appraisal  data  into  appraisal  results.  These  tools  may  be  useful  in  real 
time  as  appraisal  results  are  presented.  Strategies  for  presenting  and 
packaging  the  results  should  leverage  presentation  and  documentation 
techniques  that  best  suit  the  audience. 


It  is  highly  recommended  that  the  attendance  at  the  final  briefing  (if  one  is 
held)  be  recorded.  Significant  absenteeism  of  key  stakeholders  is  likely  to  be 
an  indication  of  risk  for  future  success  in  addressing  the  appraisal  findings. 


The  required  elements  of  appraisal  results  are  specified  in  the  activity 
description  found  here,  and  a  checklist  can  support  verification  that  these 
elements  are  present.  Validation  of  this  activity  can  only  occur  after  the 
appraisal  is  complete. 


•  Final  findings 

•  Final  report  (if  requested) 

•  Recommendations  report  (if  requested) 


If  the  method  is  being  used  as  part  of  a  supplier  selection  process,  there  may 
be  acquisition  regulations  or  limitations  that  constrain  the  mechanisms  used  to 
deliver  appraisal  results  to  the  appraised  organization. 

In  some  internal  process  improvement  usage  of  the  method,  the  executive 
session  may  be  tailored  out.  The  appraisal  sponsor  should  make  this  decision, 
with  the  full  involvement  of  the  appraisal  team  leader. 


Continued  on  next  page 
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3.1  Deliver  Appraisal  Results  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


Upon  completion  of  the  Generate  Appraisal  Results  process,  the  ratings  and 
findings  generated  are  used  to  prepare  and  deliver  the  final  appraisal  results  to 
the  appraisal  sponsor  and  organizational  unit.  The  appraisal  results  become 
part  of  the  Appraisal  Record,  which  is  discussed  in  process  3.2,  Package  and 
Archive  Appraisal  Assets. 


The  final  findings  contain  the  validated  strengths,  weaknesses,  and  ratings  (as 
defined  by  the  appraisal  plan),  reflecting  the  organizational  process  capability 
and/or  maturity  level  for  PAs  within  the  appraisal  scope.  Other  appraisal 
outputs,  as  requested  by  the  appraisal  sponsor  and  documented  in  the 
appraisal  plan,  are  generated  and  provided.  Optionally,  a  separate  executive 
session  may  also  be  held  to  clarify,  and  discuss  the  appraisal  results  from  a 
senior  management  perspective  that  facilitates  decision  making.  Plans  are 
established  for  acting  upon  the  appraisal  results. 
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3.1.1  Present  Final  Findings 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  final  findings  contain  a  summary  of  the  strengths  and  weaknesses  for 
each  PA  within  the  appraisal  scope,  as  well  as  additional  information  that 
provides  context  for  the  findings.  The  generation  of  the  findings  is  addressed 
in  activity  2.4.1,  Derive  Findings  and  Rate  Goals;  this  activity  relates  to  the 
presentation  of  these  findings  to  the  appraisal  sponsor  and  appraised 
organization.  The  presentation  may  be  in  a  summarized  form,  with  the 
detailed  findings  provided  as  backup  information,  and  is  often  presented  using 
view  graphs  in  a  meeting  room  or  auditorium. 

In  addition  to  the  final  findings,  a  draft  ADS  summarizing  the  results  of  the 
appraisal  is  provided  to  the  appraisal  sponsor. 


•  Provide  appraisal  final  findings  to  the  appraisal  sponsor  and  the 
organizational  unit. 

•  Provide  an  ADS  to  the  appraisal  sponsor  summarizing  the  appraisal 
results  and  conditions  under  which  the  appraisal  was  performed. 


Required  elements  of  the  final  findings  include 

•  summary  of  the  appraisal  process 

•  findings  (summary  of  strengths  and  weaknesses) 

Appraisal  team  consensus  must  be  obtained  on  the  wording  of  -the  final 
findings,  to  ensure  that  the  whole  team  supports  the  accuracy  of  the  described 
appraisal  results. 

The  team,  when  delivering  the  final  findings,  must  adhere  to  some  important 
principles: 

•  If  a  model  component  is  reported  as  Unsatisfied,  the  corresponding 
findings  of  weaknesses  that  caused  the  team  to  make  that  judgment  must 
also  be  reported. 

•  Confidentiality  and  non-attribution  principles  apply  to  statements  made  in 
the  presentation  of  final  findings. 

The  ADS  is  a  summary  statement  describing  the  appraisal  results  that  includes 
the  conditions  and  constraints  under  which  the  appraisal  was  performed.  It 
contains  information  considered  essential  to  adequately  interpret  the  meaning 
of  assigned  maturity  level  or  capability  level  ratings.  The  ADS  is  prepared  by 
the  appraisal  team  leader  and  provided  to  the  appraisal  sponsor.  Otherwise  the 
appraisal  team  leader  delivers  the  ADS  to  the  sponsor  as  a  separate  document. 

A  detailed  description  of  the  ADS  contents  is  provided  in  Appendix  A.  The 
ADS  is  considered  a  draft  at  this  stage  of  the  appraisal  process,  in  that  the 
ADS  must  also  contain  an  affirmation  that  all  appraisal  requirements  have 
been  satisfied,  which  cannot  be  claimed  until  the  completion  of  all  appraisal 
activities. 
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3.1 .1  Present  Final  Findings  (continued) 


Optional 

Practices 


Optional  elements  of  the  final  findings  include 

•  ratings 

•  improvement  activities 

•  recommended  actions 

•  schedule  of  major  upcoming  events  (e.g.,  appraisal  report, 
recommendations,  action  plan,  reappraisal) 

Note  that  the  generation  of  goal  ratings  by  the  appraisal  team  is  required  (as 
described  in  process  2.4,  Generate  Appraisal  Results).  However,  these  ratings 
may  be  excluded  from  the  final  findings  at  the  discretion  of  the  appraisal 
sponsor. 

A  formal  presentation  of  appraisal  results,  delivered  by  the  appraisal  team,  is 
frequently  the  final  visible  activity  for  appraisals  conducted  for  internal 
process  improvement.  The  final  findings  presentation  typically  is  delivered  in 
the  form  of  a  face-to-face  briefing  at  the  end  of  the  appraisal  on-site  period. 
Other  mechanisms  for  providing  the  appraisal  results  to  the  organizational 
unit,  such  as  written  reports,  may  be  more  practical  in  supplier  selection  or 
process  monitoring  usage  of  the  method.  The  timeframe  in  which  the 
appraisal  results  are  provided  may  also  vary,  but  the  appraisal  cannot  be 
considered  complete  until  the  final  findings  are  provided. 

The  draft  ADS  may  optionally  be  provided  during  the  executive  session(s),  if 
performed,  instead  of  at  the  conclusion  of  the  final  findings  briefing. 
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3.1.1  Present  Final  Findings  (continued) 


Implementation 

Guidance 


A  template  for  a  final  findings  briefing,  describing  its  typical  contents  and 
format,  is  provided  to  Lead  Appraisers  as  a  work  aid  by  the  CMMI  Steward. 

Findings  include  a  summary  of  strengths  and  weaknesses  determined  for  each 
PA  within  the  appraisal  reference  model  scope.  This  may  also  include  global 
findings  that  apply  across  multiple  PAs,  and  non-reference  model  findings 
that  affect  the  implementation  (positively  or  negatively)  of  associated 
processes  within  the  organizational  unit. 

Normally,  the  appraisal  team  leader  presents  the  final  findings.  In  some 
applications  of  the  method  for  internal  process  improvement,  the  team  may 
elect  to  have  an  appraisal  team  member  from  the  organizational  unit  provide 
the  briefing  to  encourage  the  acceptance  of  the  final  findings  and  ownership 
of  the  appraisal  results  for  follow-on  action. 

As  a  courtesy,  the  appraisal  team  can  consider  informing  the  appraisal 
sponsor  and/or  the  senior  site  manager  of  the  appraisal  results  prior  to 
presenting  them  publicly  in  the  final  findings  briefing.  This  may  help  them  to 
avoid  surprises  and  obtain  feedback  on  ways  to  present  the  findings  that  best 
meet  the  needs  of  the  sponsor,  appraisal  participants,  and  the  organizational 
unit.  See  activity  3.1.2,  Conduct  Executive  Session(s)  for  a  description  of 
topics  for  discussion. 

The  number  and  scope  of  findings  reported  will  affect  the  impact  of  appraisal 
results,  whether  or  not  the  team  intends  for  this  to  happen.  There  are  times 
when  providing  a  long  list  of  details  is  beneficial.  Other  times,  high-level 
summaries  are  more  appropriate. 
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3.1.2  Conduct  Executive  Session(s) 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


The  executive  session  is  an  optional  activity  that  may  be  performed  at  the 
discretion  of  the  appraisal  sponsor  or  senior  site  manager.  The  executive 
session  provides  the  appraisal  sponsor,  senior  site  manager,  and  invited  staff  a 
private  opportunity  to  (a)  discuss  with  the  appraisal  team  leader  any  issues 
with  the  appraisal,  (b)  obtain  clarification  of  the  appraisal  results,  (c)  confirm 
understanding  of  the  process  issues,  and  (d)  provide  guidance  regarding  focus, 
timing,  and  priorities  of  the  recommendations  report  and  follow-on  activities. 


None.  If  the  option  is  selected,  hold  a  private  meeting  between  the  appraisal 
team  leader  and  the  sponsor,  including  any  participants  invited  by  the  sponsor. 


If  an  executive  session  is  conducted,  the  confidentiality  and  non-attribution  of 
data  sources  must  be  maintained. 

Multiple  sessions  may  be  held  if  necessary,  targeted  at  the  information  needs 
of  the  executive  audience. 
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3.1 .2  Conduct  Executive  Session(s)  (continued) 


Optional  Attendance  by  the  entire  appraisal  team  at  the  executive  sessions  is  a  tailoring 

Practices  option. 

The  executive  session  is  also  an  appropriate  opportunity  to  review  appraisal 
performance  with  the  appraisal  sponsor  and/or  senior  site  manager,  and 
planned  versus  actual  execution  of  the  appraisal  plan,  including  method 
tailoring.  This  provides  additional  input  on  the  appropriate  expectations  for 
interpreting  and  handling  the  appraisal  results. 

The  draft  ADS  may  optionally  be  provided  during  the  executive  session 
instead  of  at  the  conclusion  of  the  final  findings  briefing  as  discussed  in 
activity  3.1.1,  Present  Final  Findings. 

The  intent  of  the  executive  sessions  is  to  ensure  that  the  appraisal  sponsor 
and/or  the  senior  site  manager  have  a  sound  understanding  of  the  appraisal 
results.  Any  feedback  obtained  from  these  executive  sessions  should  be 
recorded.  All  rules  for  confidentiality  and  non-attribution  are  still  in  effect. 

♦ 


Implementation 

Guidance 
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3.1 .3  Plan  for  Next  Steps 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 

Optional 

Practices 

Impiementation 

Guidance 


Impiementation 

Guidance 

Process 
Improvement 
Action  Planning 


Following  the  delivery  of  the  appraisal  results,  a  plan  for  follow-on  activities 
is  determined.  The  planned  follow-on  activities  are  typically  defined  in  the 
appraisal  plan,  reflecting  sponsor  requests  for  additional  appraisal  tasks  and 
products  necessary  to  meet  appraisal  objectives,  or  for  a  commitment  to  take 
action  upon  the  appraisal  results.  Follow-on  activities  may  include 

•  development  of  a  final  report 

•  development  of  a  recommendations  report  or  briefing 

•  generation  or  update  of  a  process  improvement  plan 


None. 


None. 


Planning  for  next  steps  is  an  optional,  but  recommended,  appraisal  activity. 


Planning  for  next  steps  includes  optional  activities  such  as 

•  development  of  a  final  report  by  the  appraisal  team,  summarizing  the 
appraisal  results  for  delivery  to  the  appraisal  sponsor 

•  submission  of  appraisal  team  recommendations  for  action  upon  the 
appraisal  findings 

•  generation  of  a  process  improvement  action  plan  for  the  organizational 
unit  to  act  upon  the  appraisal  findings 

In  addition  to  specifying  the  activities  to  be  performed,  these  plans  usually 
include  the  assignment  of  responsibility,  schedule,  and  estimated  resources  for 
the  implementation  of  the  follow-on*  actions.  The  plans  established  can  be 
used  to  track  the  progress  of  the  follow-on  activities  over  time. 


Findings  and  recommendations  from  the  appraisal  team  can  be  used  by  the 
organizational  unit  to  establish  action  plans  for  process  improvement.  This  is 
an  optional  output  most  often  used  in  internal  process  improvement  or 
process-monitoring  applications  of  the  appraisal  method. 

Recommendations  often  include  a  prioritized  list  of  improvement  activities, 
including  the  development  of  an  improvement  plan  that  defines  the  tasks, 
schedules,  and  resources  necessary  for  implementation. 

Follow-on  appraisals  are  usually  performed  to  verify  improvement  progress. 
This  might  include  a  combination  of  Class  A,  Class  B,  and  Class  C  appraisals 
(refer  to  the  ARC  for  additional  details). 
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3.1 .3  Plan  for  Next  Steps  (continued) 


Implementation 

Guidance 

Final  Report 


Implementation 

Guidance 

Recommend¬ 
ations  Report 


The  purpose  of  the  final  report  is  to  provide  details  or  explanations  beyond 
what  was  contained  in  the  final  findings.  The  generation  of  an  appraisal  final 
report  is  an  optional  activity  that,  if  requested  by  the  appraisal  sponsor, 
documents  the  execution  of  the  appraisal,  contains  detailed  appraisal  findings, 
and  forms  a  basis  for  action  planning.  This  baseline  is  used  for  subsequent 
reports  and  follow-on  actions,  and  also  may  be  an  input  for  use  in  subsequent 
appraisals. 

Items  contained  or  referenced  in  the  final  report,  either  in  their  entirety  or  as  a 
subset,  might  include 

•  executive  summary  of  the  appraisal  process  and  results 

•  appraisal  input  (see  process  1.1) 

•  appraisal  plan  (see  process  1.2) 

•  appraisal  record  (see  process  3.2) 

The  final  report  should  be  completed  as  soon  after  the  appraisal  as  possible, 
preferably  within  four  weeks.  The  appraisal  team  leader  usually  generates  the 
final  report;  other  team  members  may  also  contribute. 

The  format  and  content  of  the  final  report  may  vary  according  to  its  intended 
use  by  the  appraisal  sponsor.  In  its  simplest  form,  this  could  be  a  set  of  notes 
annotated  to  the  final  findings,  elaborating  on  some  aspect  of  the  findings  or 
capturing  essential  comments  or  recommendations  from  the  appraisal  team. 


If  requested  by  the  appraisal  sponsor,  appraisal  team  recommendations  for 
taking  action  on  the  appraisal  results  can  be  provided.  These 
recommendations  can  provide  momentum  to  the  appraisal  follow-up  by 
serving  as  a  link  between  the  appraisal  findings  and  subsequent  decision 
making  or  action  plans.  The  emphasis  of  these  recommendations  depends  on 
the  appraisal  sponsor’s  objectives  and  planned  use  of  the  appraisal  results,  as 
defined  in  the  appraisal  input.  This  can  vary  widely  based  on  the  context  in 
which  the  appraisal  method  is  applied  (e.g.,  internal  process  improvement, 
supplier  selection,  process  monitoring). 

The  recommendations  report  should  be  completed  as  soon  after  the  appraisal 
on-site  period  as  possible.  Depending  on  the  nature,  complexity,  and  use  of 
the  recommendations,  this  may  take  as  long  as  two  months  to  produce. 

Rather  than  generate  a  separate  recommendations  report,  a  common 
alternative  is  to  include  these  recommendations  in  the  final  report. 

It  is  important  to  consider  the  possibility  that  the  expertise  needed  for  making 
the  appropriate  recommendations  may  be  beyond  the  level  of  expertise 
reflected  on  the  team. 


♦ 
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3.2  Package  and  Archive  Appraisal  Assets 


Purpose 
Entry  Criteria 

Inputs 

Activities 

Outputs 

Outconie 

Exit  Criteria 


Preserve  important  data  and  records  from  the  appraisal,  and  dispose  of 
sensitive  materials  in  an  appropriate  manner. 


•  Appraisal  has  been  conducted. 

•  Results  have  been  delivered  to  the  sponsor. 

•  All  appropriate  data  have  been  collected  and  retained  during  the  appraisal. 


•  Appraisal  data 

appraisal  input 
appraisal  plan 
final  findings 
objective  evidence 

•  Appraisal  team  artifacts 

notes 

documented  practice  implementation  gaps 
preliminary  findings 
^  document  library 


3.2. 1  Collect  Lessons  Learned 

3.2.2  Generate  Appraisal  Record 

3.2.3  Provide  Appraisal  Feedback  to  CMMI  Steward 

3.2.4  Archive  an^or  Dispose  of  Key  Artifacts 


•  Appraisal  Record 

•  Completed  forms  and  checklists 

•  Sanitized  data  (as  appropriate  and  agreed  upon  during  planning) 

•  Lessons  learned  (appraisal  team,  organization) 


Data  and  artifacts  are  appropriately  archived  or  destroyed.  The  team  has 
captured  lessons  and  data  to  help  improve  the  appraisal  process.  Requirements 
for  providing  appraisal  artifacts  to  stakeholders  and  the  CMMI  Steward  are 
met. 


•  Appraisal  assets  are  baselined  and  archived. 

•  Required  reports  are  delivered  to  the  appropriate  stakeholders. 

•  Artifacts  containing  sensitive  information  are  disposed  of  in  an 
appropriate  manner. 
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3.2  Package  and  Archive  Appraisal  Assets  (continued) 


Key  Points 

Tools  and 
Techniques 

Metrics 

Verification  and 
Validation 

Records 

Tailoring 


Protect  the  confidentiality  of  sensitive  data  while  distributing  and  archiving 
appraisal  assets.  Bundle  related  information  together  whenever  appropriate. 


The  use  of  electronic  (database)  tools  for  managing  appraisal  data  often 
provides  assistance  in  ensuring  the  integrity  of  baselines,  as  well  as 
repackaging  information  for  archival  purposes.  Electronic  tools  allow  the 
Lead  Appraiser  to  remove  traceability  information  so  that  data  can  be 
provided  to  the  appropriate  people  while  preserving  the  anonymity  of  the  data 
sources. 

Electronic  tools  also  support  the  submission  of  appraisal  data  to  the  CMMI 
Steward.  This  reduces  the  administrative  burden,  and  will  facilitate  the 
analysis  of  appraisal  method  performance  data.  These  tools  also  provide 
feedback  on  the  consolidated  analysis  results  to  the  appraisal  community. 


While  archiving  and  reporting  the  metrics  associated  with  the  conduct  of  the 
appraisal  is  an  important  element  of  this  activity,  the  metrics  associated  with 
the  conduct  of  this  activity  itself  are  limited.  The  effort  and  calendar  time 
consumed  are  collected  and  compared  to  the  plan.  Some  appraisal  team 
leaders  will  choose  to  maintain  personal  metrics  associated  with  the  artifacts 
described  in  this  activity. 


The  Lead  Appraiser  Requirements  Checklist  guides  the  verification  of  the  list 
of  artifacts  provided  to  the  CMMI  Steward.  Validation  is  provided  by  the 
CMMI  Steward  upon  receipt  of  the  appraisal  record. 


•  Appraisal  Record 

•  Lessons  Learned 


The  usage  mode  and  constraints  of  the  appraisal,  as  well  as  the  sensitivity  of 
the  data  and  planned  use  of  appraisal  results,  may  greatly  affect  the  degree  to 
which  appraisal  data  is  retained,  sanitized,  or  discarded. 
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3.2  Package  and  Archive  Appraisal  Assets  (continued) 


As  the  final  process  in  the  appraisal,  this  process  is  about  collecting, 
packaging,  and  archiving  those  results  and  artifacts  produced  by  previous 
processes  that  must  become  part  of  the  appraisal  record.  Most  notably,  this 
includes  the  appraisal  input,  appraisal  plan,  and  appraisal  results. 
Additionally,  sensitive  or  proprietary  data  produced  by  other  appraisal 
processes  must  be  returned  to  the  organizational  unit  or  destroyed. 

Summary  of  This  process  performs  the  data  collection,  data  management,  and  reporting 

Activities  activities  necessary  to  close  out  the  appraisal.  Data  collected  throughout  the 

appraisal  is  consolidated  and  baselined,  becoming  a  permanent  part  of  the 
appraisal  record. 

♦ 


Interfaces 
with  Other 
Processes 
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3.2.1  Collect  Lessons  Learned 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


As  one  of  the  final  activities  in  wrapping  up  an  appraisal,  teams  typically 
record  lessons  learned  from  their  experience.  The  purpose  of  these  lessons 
learned  is  to  document  what  went  right,  what  went  wrong,  and  any 
suggestions  or  recommendations  for  improving  the  method  or  its  execution. 
The  collection  of  lessons  learned  is  a  recommended  activity  for  the 
improvement  of  future  appraisals,  but  is  not  a  method  requirement. 


None. 


Lessons  learned  must  adhere  to  the  same  principles  of  confidentiality  and 
non-attribution  applicable  to  other  appraisal  results. 
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3.2.1  Collect  Lessons  Learned  (continued) 


All  practices  related  to  the  collection  of  lessons  learned  are  optional,  but 
recommended.  If  the  team  has  identified  potential  improvements  to  elements 
of  the  CMMI  Product  Suite  (reference  model,  appraisal  method,  and  training 
materials),  these  can  be  submitted  as  change  requests  to  the  CMMI  Steward. 

Implementation  Capturing  lessons  learned  is  often  done  as  a  group  at  the  end  of  the  appraisal. 
Guidance  while  the  appraisal  activities  are  fresh  in  team  members’  minds.  This  can  be 

supplemented  with  additional  inputs  from  team  members  upon  further 
reflection,  if  necessary.  Appraisal  team  leaders  forward  these  aggregate 
lessons  learned,  as  appropriate,  to  various  stakeholders,  but  always  to  the 
other  team  members.  Team  leaders  and  members  often  maintain  summary 
lists  of  appraisal  best  practices  and  lessons  learned  as  a  mechanism  for 
continuous  learning  and  improvement,  and  these  are  used  as  a  resource  for 
planning  subsequent  appraisals. 

♦ 


Optional 

Practices 
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3.2.2  Generate  Appraisal  Record 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Appraisal  data  collected  throughout  the  appraisal  is  aggregated  and 
summarized  into  a  permanent  record  documenting  the  appraisal  conduct  and 
results.  The  appraisal  record  is  delivered  to  the  appraisal  sponsor  for  retention. 


•  Collect  and  baseline  appraisal  data  that  becomes  part  of  the  permanent 
records  provided  to  appraisal  stakeholders. 

•  Document  the  satisfaction  of  all  SCAMPI  requirements. 

•  Generate  the  appraisal  record  from  baselined  planning  and  execution  data 
collected  throughout  the  appraisal. 

•  Deliver  the  appraisal  record  to  the  appraisal  sponsor. 


Required  contents  of  the  appraisal  record  include  the  following; 

•  dates  of  the  appraisal 

•  appraisal  input 

•  appraisal  plan 

•  objective  evidence,  or  identification  thereof,  sufficient  to  substantiate 
goal-rating  judgments 

•  characterizations  of  practice  implementation  determined  at  the 
instantiation  level  and  aggregated  at  the  organizational  unit  level 

•  identification  of  the  appraisal  method  (and  version)  used  along  with  any 
tailoring  options 

•  final  findings 

•  all  ratings  rendered  during  the  appraisal  (goals,  PAs,  and  maturity  or 
capability  levels) 

•  ADS 

•  the  set  of  15504  process  profiles  resulting  from  the  appraisal  (if  requested 
by  the  appraisal  sponsor) 

Depending  on  the  recipient  and  intended  usage,  appraisal  data  may  be  subject 
to  being  sanitized  or  edited  in  order  to  comply  with  rules  for  non-attribution, 
confidentiality,  protection  of  proprietary  information,  and  applicable  laws, 
regulations,  or  standards  (e.g.,  acquisition  regulations  or  security 
classification).  Recipients  are  expected  to  place  the  appropriate  limitations  on 
the  access  and  use  of  the  provided  appraisal  data. 

The  appraisal  team  leader  documents  in  the  ADS  that  all  SCAMPI 
requirements  were  satisfied. 


Continued  on  next  page 


Page  11-140 


CMU/SEI-2001-HB-001 


3.2.2  Generate  Appraisal  Record  (continued) 


Optional  The  appraisal  record  should  also  contain  any  additional  outputs  requested  by 

Practices  the  appraisal  sponsor,  as  agreed  to  during  appraisal  planning  and  documented 

in  the  appraisal  input. 


Implementation  The  actual  objective  evidence  (artifacts  or  portions  of  artifacts)  need  not  be 
Guidance  part  of  the  appraisal  record,  but  an  identification  of  the  objective  evidence  is 

required.  This  may  be  implemented  by  providing  the  PIIs  that  were  used  as 
the  basis  for  characterizing  practice  implementation  decisions. 

Guidance  on  the  protection  of  appraisal  data  can  be  summarized  based  on  the 
recipient  of  the  data  as  follows: 

•  appraisal  sponsor:  replacement  of  specific  sources  (persons,  projects)  with 
non-attributable,  general  identifiers  (e.g.,  numeric  codes  assigned  to 
projects,  roles,  or  data-gathering  sessions).  If  the  sponsor  is  separate  from 
the  appraised  organization  (e.g.,  in  the  case  of  a  supplier  selection 
context),  there  may  be  situations  where  confidential  or  proprietary  data 
relating  to  the  appraised  organization  must  be  removed. 

•  CMMI  Steward:  same  as  for  appraisal  sponsor,  for  data  that  is  shared  by 
both.  For  data  that  is  provided  only  to  the  CMMI  Steward,  the  data 
collection  vehicles  (e.g.,  forms)  are  already  designed  to  observe  non- 
attiibution  and  confidentiality  rules.  Additionally,  supplied  data  may  be 
subject  to  further  sanitization  to  comply  with  acquisition  or  security- 
related  restrictions. 

•  senior  site  manager:  in  cases  where  the  appraised  organizational  unit  is 
separate  from  the  appraisal  sponsor,  the  appraised  organization  is 
typically  provided  only  with  appraisal  results  and  not  data  related  to 
planning  and  decision  making,  or  data  that  makes  use  of  the  results. 
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3.2.3  Provide  Appraisal  Feedback  to  CMMI  Steward 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


Optional 

Practices 

Implementation 

Guidance 


Appraisal  data  required  by  the  CMMI  Steward  is  collected  and  reported.  This 
includes  a  subset  of  the  contents  of  the  appraisal  record,  as  well  other  data 
used  by  the  Steward  to  aggregate  and  analyze  appraisal  performance  data  for 
reporting  to  the  community  and  monitoring  the  quality  of  performed 
appraisals. 


Submit  the  completed  appraisal  report  as  required  by  the  CMMI  Steward. 


The  CMMI  Steward  defines  the  specific  set  of  data  required  for  submission  at 
the  completion  of  an  appraisal.  Submission  of  the  appraisal  report  is  required 
for  the  appraisal  to  be  recorded  in  the  Steward’s  database  of  appraisal  results. 
This  is  also  a  requirement  established  by  the  Steward  to  maintain  Lead 
Appraiser  authorization. 


If  the  objective  evidence  is  available  in  electronic  form,  it  can  be  included  as 
part  of  the  appraisal  report  submitted  to  the  CMMI  Steward. 


The  appraisal  team  leader  is  responsible  for  ensuring  that  appraisal  feedback 
required  by  the  CMMI  Steward  is  collected  and  reported.  The  CMMI 
Steward,  as  custodian  of  the  product  suite  and  the  Appraiser  Program,  has 
several  objectives  in  seeking  appraisal  feedback: 

•  characterization  of  the  state  of  the  practice  in  the  appraisal  community, 
for  the  collection  and  distribution  of  effective  appraisal  techniques 

•  analysis  of  reported  appraisal  data  to  obtain  an  understanding  of  appraisal 
performance  for  continuous  improvement 

•  quality  control  within  the  Appraiser  Program,  to  ensure  a  high  level  of 
confidence  in  the  accuracy  of  appraisal  results 

Feedback  is  provided  periodically  to  the  community  on  summarized  results 
determined  from  appraisal  data  collected. 

The  format  and  mechanisms  for  the  submission  of  this  data  are  established  by 
the  CMMI  Steward. 


♦ 
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3.2.4  Archive  and/or  Dispose  of  Key  Artifacts 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


Optional 

Practices 

Implementation 

Guidance 


After  the  various  reports  are  delivered  to  the  appropriate  stakeholders  and  the 
appraisal  assets  have  been  baselined,  the  team  leader  is  responsible  for 
properly  archiving  and/or  disposing  of  the  appraisal  data,  in  accordance  with 
agreements  made  with  the  sponsor  and  documented  in  the  appraisal  input.  The 
team  librarian  (if  one  is  used)  ensures  that  all  organization-provided 
documentation  and  objective  evidence  is  returned  or  disposed  of  properly. 
Any  remaining  team  artifacts  or  notes  are  disposed  of  properly. 


•  Archive  or  dispose  of  key  artifacts  collected  by  the  appraisal  team. 

•  Return  objective  evidence  provided  by  the  organizational  unit. 


In  all  usage  modes  of  SCAMPI,  strict  non-attribution  policies  apply. 
Confidentiality  and  non-disclosure  agreements  established  with  the  appraisal 
team  members  remain  in  effect. 


None. 


How  the  records  will  be  preserved  or  disposed  of  is  dependent  on  the  usage 
mode  of  the  method  and  the  appraisal  objectives  that  shape  the  current 
application.  Confidentiality  rules  may  differ  by  application.  In  a  supplier 
selection  usage,  the  results  are  not  proprietary  in  that  the  sponsor  is  not  a 
member  of  the  appraised  organization.  However,  results  are  only  known  to  the 
sponsor  and  the  recipient;  competing  organizations  do  not  see  the  results. 
Confidentiality  of  results  can  be  characterized  as  one  of  the  following: 

•  known  only  to  the  recipient  organization 

•  known  to  the  recipient  and  sponsor,  when  they  are  from  different 
organizations 

•  known  to  anyone 

The  sponsor  is  solely  responsible  for  determining  the  confidentiality  with 
which  the  appraisal  results  will  be  maintained.  The  non-attribution  of  data  to 
specific  individuals  is  the  responsibility  of  the  appraisal  team.  The  recipient 
organization,  if  the  sponsor  agrees  and  it  is  planned  for,  may  always  choose  to 
make  the  results  known  outside  the  organization.  At  a  high  level,  this  might 
be  done  for  marketing  and  public  relations  reasons.  Disclosures  of  results 
include  the  context  and  constraints  under  which  the  appraisal  was  performed 
(e.g.,  reference  model  scope,  organizational  scope),  as  defined  by  the  ADS 
described  in  process  3.1,  Deliver  Appraisal  Results. 

Any  annotations  related  to  the  objective  evidence  provided  to  the  organization 
by  the  appraisal  team  should  be  recorded  and  archived  for  use  in  process 
improvement  actions  or  for  reuse  in  subsequent  appraisals. 
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Appendix  A  Appraisal  Disclosure 

Statement 


The  Appraisal  Disclosure  Statement  (ADS)  provides  information  considered  essential  to  ade¬ 
quately  interpret  the  meaning  of  maturity  level  or  capability  level  ratings  resulting  from  a 
CMMI  Class  A  appraisal. 

The  ADS  is  prepared  by  the  appraisal  team  leader  and  provided  to  the  appraisal  sponsor  at  the 
conclusion  of  the  appraisal.  If  the  final  findings  briefing  reports  the  appraisal  ratings,  the  ve¬ 
hicle  for  reporting  the  ratings  must  be  the  ADS.  Otherwise  the  appraisal  team  leader  delivers 
the  ADS  to  the  sponsor  as  a  separate  document. 

ADS  Content 

The  ADS  consists  of  the  following  information: 

•  identification  of  appraisal  sponsor  and  sponsor’s  organizational  affiliation 

•  identification  of  appraisal  team  leader  and  appraisal  team  members  and  their  organiza¬ 
tional  affiliations 

•  identification  of  organizational  unit  appraised  (the  unit  to  which  the  ratings  are  applicable 
and  the  domains  examined,  as  defined  in  the  appraisal  plan) 

•  identification  of  CMMI  model  (version,  representation,  and  domains) 

•  identification  of  appraisal  method  (name  and  version) 

•  itemization  of  process  areas  rated  and  process  areas  not  rated 

•  maturity  level  and/or  capability  level  ratings  assigned 

•  dates  of  on-site  activity 

•  date  of  issuance  of  ADS 

•  statement  affirming  that  all  SCAMPI  requirements  were  met 

•  signature  of  appraisal  team  leader  (at  a  minimum);  those  of  appraisal  team  members  and 
appraisal  sponsor  are  optional 
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Appendix  B  The  Role  of  Practice 

Implementation  Indicators  in 
Verifying  Practice 
Implementation 


Purpose 

This  appendix  provides  a  conceptual  overview  of  the  process  of  verifying  practice  implemen¬ 
tation  and  the  role  of  Practice  Implementation  Indicators  in  that  process.  Verification  of  prac¬ 
tice  implementation  is  an  essential  element  of  appraising  the  implementation  of  processes 
relative  to  models  of  best  practices  such  as  CMMI. 

Verifying  Practice  Implementation 

In  this  discussion,  verifying  CMMI  practice  implementation  means  the  substantiation  of  prac¬ 
tice  implementation  based  on  a  review  of  objective  evidence.  For  example,  one  might  inquire 
as  to  whether  a  project-specific  practice  is  implemented  within  a  project.  Alternatively,  one 
might  inquire  as  to  whether  an  organization-specific  practice  is  implemented  within  an  or¬ 
ganization. 

Having  a  well-defined  approach  for  verifying  practice  implementation  is  of  critical  impor¬ 
tance  from  several  perspectives.  For  the  process  improvement  sponsor,  it  provides  some  as¬ 
surance  that  the  resources  applied  to  the  improvement  effort  will  result  in  the  desired  out¬ 
come  and  that  the  resultant  benefits  can  therefore  be  expected.  For  process  improvement 
agents  or  champions,  it  enables  them  to  know  when  they  have  succeeded  with  the  implemen¬ 
tation  activity,  and  to  informally  monitor  whether  the  practice  continues  to  be  implemented 
over  time.  For  appraisal  teams,  a  well-defined  verification  approach  is  essential  for  determin¬ 
ing  what  capability  level  or  maturity  level  ratings  are  warranted.  CMMI  process  area  goal 
satisfaction  is  predicated  on  implementation  of  the  relevant  specific  or  generic  practices  (or 
acceptable  altematives)\  Hence  verification  of  practice  implementation  is  a  crucial  appraisal 
task. 


*  See  “Required,  Expected,  and  Informative  Components”  in  Chapter  2  of  the  CMMI  model  that 
you  are  using. 
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Practice  Implementation  Indicators 

’The  fundamental  idea  of  Practice  Implementation  Indicators  (PIIs)  is  quite  simple  and 
broadly  applicable  to  any  practice  or  activity.  It  is  based  on  the  obvious  fact  that  the  conduct 
of  an  activity  or  the  implementation  of  a  practice  will  result  in  ‘‘footprints” — evidence  that 
the  activity  was  conducted  or  the  practice  was  implemented. 

For  example,  if  one  balances  one’s  checkbook  at  the  end  of  the  month,  there  are  several  po¬ 
tential  ways  to  confirm  that  this  activity  has  indeed  taken  place.  First,  the  person  who  en¬ 
gaged  in  the  checkbook  balancing  activity  can  affirm  that  this  activity  was  conducted.  Sec¬ 
ond,  there  will  likely  be  an  entry  in  the  checkbook  register  for  each  check  or  transaction  to 
indicate  that  it  matches  with  a  corresponding  entry  in  the  bank’s  statement.  Additional  arti¬ 
facts  could  be  identified. 

The  general  idea  is  clear:  the  actual  conduct  of  an  activity  leaves  footprints  that  provide  a 
basis  for  verification. 


PIIs  refer  to  the  footprints  that  are  the  necessary  and  unavoidable  consequence  of  practice 
implementation.  They  include  information  contained  in  artifacts  and  information  gathered 
from  interviews  with  managers  and  practitioners. 

The  Role  of  PIIs 

ARC-compliant  appraisal  methods  employ  objective  evidence  obtained  from  one  or  more 
sources  (instruments,  documents,  and  interviews).  An  appraisal  team  bases  its  decisions  about 
practice  implementation  on  examination  of  this  objective  evidence. 

Once  a  project  or  organization  has  an  understanding  of  how  its  processes  relate  to  the  CMMI 
model,  the  stage  is  set  for  capturing  the  PIIs  that  provide  the  objective  evidence  of  implementa¬ 
tion.  The  work  of  establishing  the  collection  of  PIIs  for  the  project(s)  and/or  organization  pro¬ 
vides  assurance  to  the  process  improvement  sponsor  that  the  expected  implementation  activities 
have  in  fact  resulted  in  alignment  of  the  organization’s  activities  with  the  CMMI  model. 


This  aggregation  of  objective  evidence — ^the  PIIs — is  itself  an  important  organizational  proc¬ 
ess  asset  that  has  a  number  of  potential  uses,  most  notably  providing  an  appraisal  team  a  head 
start  in  understanding  the  organization’s  implementation  of  the  CMMI  model.  This  leaves  the 
appraisal  team  the  task  of  verifying  whether  the  objective  evidence^  provided  is  adequate  for 
substantiation  of  practice  implementation,  rather  than  the  more  difficult,  error  prone,  and 

The  ARC  defines  objective  evidence  as  “qualitative  or  quantitative  information,  records,  or  state¬ 
ments  of  fact  pertaining  to  the  characteristics  of  an  item  or  service  or  to  the  existence  and  imple¬ 
mentation  of  a  process  element,  which  are  based  on  observation,  measurement,  or  test  and  are 
verifiable.” 


III-6 


CMU/SEI-2001-HB-001 


time-consuming  task  of  investigating  each  practice  to  discover  the  objective  evidence  needed 
to  substantiate  implementation. 

Both  the  appraised  organization  and  the  appraisal  team  have  a  clearer  picture  of  what  artifacts 
need  to  be  provided  to  substantiate  implementation  of  the  practices,  thereby  minimizing  the 
amount  of  further  investigation  necessary  in  the  form  of  interviews  and  additional  documen¬ 
tation  requests.  The  extent  to  which  the  appraised  organization  can  provide  this  information 
becomes  a  principal  factor  in  how  much  further  investigation  may  be  required. 

Another  benefit  of  this  approach  is  significantly  greater  reliability  and  accuracy  of  appraisal. 

The  Pll-based  approach  is  not  meant  to  turn  the  appraisal  into  a  documentation  review  exer¬ 
cise.  It  merely  allows  for  more  focused  and  effective  use  of  the  on-site  phase  and  potentially 
a  shorter  on-site  phase  than  would  otherwise  be  the  case. 

Finally,  the  PIIs  are  not  intended  to  tie  the  hands  of  model  implementers  or  process  appraisal 
teams.  The  primary  value  of  the  PIIs  lies  in  making  explicit  what  has  heretofore  been  implicit 
and  therefore  subject  to  wide  variations  in  interpretation  and  understanding.  Over  time,  shar¬ 
ing  of  PIIs  will  result  in  a  set  of  practice  implementation  scenarios  (e.g.,  small,  medium,  and 
large  organizations  or  projects)  and  a  standard  set  of  PIIs  that  could  be  used  as  a  starting 
point  for  further  customization.  The  particular  process  implementation  context  and  the  specif¬ 
ics  of  the  project  would  determine  which  of  the  indicators  make  sense  for  that  implementa¬ 
tion.  Appraisal  teams  would  be  obliged  to  inquire  into  the  existence  of  the  agreed-upon  indi¬ 
cators,  while  still  having  the  freedom  to  make  judgments  based  on  the  facts  and 
circumstances  of  the  implementation. 

A  standard  set  of  PIIs  could  establish  norms  within  which  most  implementations  will  fall, 
thereby  allowing  efficiencies  to  be  realized  in  implementation  and  appraisal,  while  at  the 
same  time  recognizing  that  alternative  implementations  may  be  possible  using  alternative 
practices. 

PII  Components 

PIIs  have  two  components  or  dimensions:  an  objective  evidence  component  and  a  practice 
implementation  type  component.  The  objective  evidence  component  refers  to  the  form  of  the 
objective  evidence.  The  practice  implementation  type  component  deals  with  the  significance 
of  the  objective  evidence  in  relation  to  practice  implementation. 
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Forms  of  Objective  Evidence 

An  appraisal  team  bases  its  decisions  about  practice  implementation  on  the  existence  of  ob¬ 
jective  evidence  available  to  it.  This  objective  evidence  can  take  on  one  or  more  of  the  fol¬ 
lowing  forms: 

•  artifacts 

-  work  products,  which  are  the  explicit  intended  consequences  of  practice  inplementation 

-  artifacts  that  are  incidental  to,  but  indicative  of,  practice  implementation 

•  affirmations 

-  written  or  oral  statements  indicative  of  practice  implementation  from  practitioners 
who  carry  out  the  activities  relevant  to  the  practice  or  from  suppliers,  customers,  or 
other  stakeholders  in  the  practice 

-  demonstrations  or  presentations  (e.g.,  the  demonstration  of  capability  of  a  tool  or 
other  mechanism  as  it  relates  to  the  implementation  of  a  practice,  or  a  presentation 
explaining  some  aspect  of  the  organization  or  project) 

Note  that  there  is  not  a  strong  distinction  made  in  the  model  between  artifacts  and  work 
products  (see  Chapter  3  in  the  model  for  an  explanation  of  how  “work  product”  is  used).  As 
used  in  the  context  of  PIIs,  work  product  refers  to  an  artifact  that  is  either  explicitly  men¬ 
tioned  in  the  statement  of  a  CMMI  practice  or  whose  absence  would  be  a  strong  indictor  of 
incomplete  or  inadequate  practice  implementation.  The  weaker  term  “artifact”  is  used  in  the 
context  of  PHs  to  refer  to  an  artifact  whose  existence  is  incidental  to  (i.e.,  a  side-effect  of)  the 
accomplishment  of  the  main  intent  of  the  practice. 


Types  of  PIIs 

Using  the  above  discussion  as  the  framework,  it  is  now  possible  to  itemize  the  types  of  PIIs 
that  might  be  present  as  a  consequence  of  practice  implementation.  Table  III-l  shows  PII 
types,  which  collectively  provide  coverage  for  any  CMMI  practice.  Each  type  is  described  in 
more  detail  below. 


Table  III-1:  PII  Types 


PII 

Type 

Objective 
Evidence  Form 

Generic  Description 

Direct 

Artifact  (work 
product) 

Work  product(s)  that  reflect  (document  the  information  content  of) 
the  establishment  of  {insert  text  from  practice  statement  that  de¬ 
scribes  object  of  practice  enactment}. 

Indirect 

Artifact 

Artifact(s)  that  are  an  indirect  consequence  (or  side-effect)  of  the 
effort  required  to  (insert  text  from  practice  statement  that  describes 
object  of  practice  enactment). 

Direct 

Affirmation 

Affirmations  from  individuals  who  participated  in  or  contributed  to 
(insert  text  from  practice  statement  that  describes  object  of  practice 
enactment)  OR  affirmations  from  individuals  who  are  users  of  (or 
who  can  substantiate  use  of)  (insert  text  from  practice  statement 
that  describes  object  of  practice  enactment). 
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Direct  Artifact 


This  PII  type  is  relevant  when  establishment  of  a  work  product  is  an  integral  part  of  practice 
implementation.  Sometimes  this  is  explicit,  as  in  “Establish  and  maintain  process  action  plans 
to  address  improvements  to  the  organization’s  processes  and  related  process  assets”  (OPF  SP 
2.1-1).  In  other  instances,  it  is  not  explicit,  although  it  would  be  difficult  to  imagine  practice 
implementation  without  the  presence  of  one  or  more  work  products  being  produced.  In  most 
cases,  the  model  document  already  identifies  these  work  products. 

Indirect  Artifact 

This  PII  type  applies  to  artifacts  that  are  produced  as  a  natural  consequence  of  practice  en¬ 
actment.  The  difference  between  this  and  a  direct  artifact  PII  is  that  this  type  applies  to  arti¬ 
facts  that  are  an  indirect  consequence  or  side-effect  of  practice  enactment.  For  this  reason, 
artifacts  that  are  relevant  to  this  PII  will  vary  widely  and  will  tend  to  be  implementation- 
specific.  This  indicator  type  is  especially  useful  when  there  may  be  doubts  about  whether  the 
intent  of  the  practice  has  been  met  (e.g.,  a  work  product  exists  but  there  is  no  indication  of 
where  it  came  from  or  who  developed  it). 

Direct  Affirmation 

This  PII  type  refers  to  either  information  obtained  via  interviews  of  individuals  involved  in 
the  enactment  of  a  practice  or  of  individuals  who  are  stakeholders  (e.g.,  customers,  suppliers) 
in  the  enactment  of  a  practice.  This  type  can  also  apply  to  information  provided  in  other 
ways,  such  as  demonstrations  or  presentations. 

PII  Descriptions 

A  PII  Description  (PHD)  is  a  structure  or  schema  defined  to  provide  a  repository  for  the  PII 
information.  Table  III-2  shows  an  example  of  such  a  structure.  Note  that  this  is  a  notional 
description  of  the  content,  not  a  physical  definition  of  the  format. 
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Table  II 1-2:  A  PHD  Schema 


Attribute 

Synopsis 

Remarks 

Practice  ID 

This  identifies  the  process  area,  goal,  and  practice 
that  the  PII  is  associated  with. 

Acronyms  found  in  the 
CMMI  models  are  used. 

PIIID 

This  identifies  the  indicator  type  and  the  form  of 
objective  evidence. 

Types  are  direct  artifact, 
indirect  artifact,  and  di¬ 
rect  affirmation. 

Description 

This  is  a  description  of  the  PII  as  applied  to  this  prac¬ 
tice. 

Examples 

These  are  examples  of  artifacts  or  affirmations  that 
would  exemplify  the  intent  of  the  PII  and/or  explora¬ 
tory  questions  (EQs)  or  “look  fors”  (LFs).  They  as¬ 
sist  assessors  in  identifying  relevant  artifacts  or  elic¬ 
iting  relevant  information. 

Aim  to  minimize  any 
overlap  with  such  infor¬ 
mation  that  is  already  in 
the  model  document. 

Organizational 

Implementation 

This  attribute  would  be  filled  in  by  the  organization 
as  part  of  its  implementation  program  and  provided 
to  the  appraisal  team  as  a  resource. 

Table  III-3  shows  an  example  PIDD  for  specific  practice  1.1-1  of  the  Project  Planning  process 
area: 


Table  III-3:  An  Example  PHD 


Attribute 

Value 

Practice  ID 

PPSP  1.1-1 

PIIID 

Direct  Artifact 

PII  Description 

Work  product(s)  that  reflect  (document  the  information  content  of)  the  estab¬ 
lishment  of  a  top-level  work  breakdown  structure  (WBS)  to  estimate  of  the  scope 
of  the  project. 

Examples 

See  Typical  Work  Products. 

Organizational 

Implementation 

{To  be  provided  by  the  organization  for  a  specific  project  implementation.} 

These  descriptions  have  a  number  of  uses  in  addition  to  their  utility  during  process  appraisal. 
They  can  be  used  during  the  model  implementation  phase,  after  model  implementation  as  a 
training  vehicle  for  new  personnel,  for  internal  monitoring  of  practice  implementation,  etc. 

Application  of  Plis  in  Model  Implementation 

The  use  of  indicators  has  significant  utility  for  an  organization  that  is  committed  to  model- 
based  process  improvement.  Typically,  organizations  will  either  implement  model  practices 
directly  or  will  ensure  that  the  practices  used  in  the  organization  effect  goal  achievement 
(through  the  mechanism  of  alternative  practices). 
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Since  models  are  necessarily  expressed  and  published  in  an  implementation-independent 
manner,  the  implementation  of  a  model  will  require  that  an  understanding  of  how  the  model 
intent  (as  expressed  though  goals,  practices,  and  other  model  material)  is  to  be  realized  in  the 
organization  be  developed,  documented,  and  operationalized.  The  model  intent  is  made  real 
through  its  impact  on  the  way  people  work;  if  there  is  no  relation  between  how  they  work  and 
the  model,  the  organization  has  not  implemented  the  model.  Thus,  having  an  understanding 
of  the  ways  in  which  implementation  of  the  model  relates  to  what  people  are  doing  in  the  or¬ 
ganization  is  a  necessary  and  unavoidable  prerequisite  to  implementing  the  model.  PIIDs 
provide  a  mechanism  by  which  the  implementation  of  a  model  practice  can  be  described. 

Application  of  Plls  in  Process  Appraisal 

During  the  course  of  process  appraisal,  the  appraisal  team’s  primary  focus  is  on  verifying 
practice  implementation.  This  is  accomplished  by  (1)  obtaining  objective  evidence  relevant  to 
the  implementation  of  a  practice,  (2)  comparing  the  objective  evidence  available  with  what  is 
expected,  and  then  (3)  making  a  determination  of  practice  implementation  based  on  the  dif¬ 
ference  between  actual  and  expected  evidence. 

The  PIIs  assist  the  appraisal  team  (as  well  as  the  implementing  organization)  with  task  1  by 
providing  a  framework  or  structure  that  makes  explicit  the  types  of  objective  evidence  that 
should  be  considered.  In  concert  with  the  CMMI  model  documentation,  this  provides  the 
model  basis  against  which  the  organization’s  actual  operating  practices  are  compared. 

Note  that  PIIs  do  not  prescribe  what  objective  evidence  must  be  present  for  practice  imple¬ 
mentation  determinations  to  be  made;  they  only  make  explicit  what  is  reasonable  for  an  ap¬ 
praisal  team  to  consider.  The  particular  circumstances  and  attributes  of  the  organizational  unit 
and/or  project  must  all  be  taken  into  consideration  when  making  determinations  of  practice 
implementation.  As  a  general  rule,  the  more  objective  evidence  and  the  more  PIIs  represented 
by  that  objective  evidence,  the  higher  the  confidence  level  that  the  practice  is  implemented. 

The  PII  structure  assists  the  appraisal  team  with  task  2  to  the  extent  that  the  team  has  agreed 
in  advance  on  the  objective  evidence  it  expects  to  see  for  each  process  instantiation  exam¬ 
ined.  In  some  cases  it  may  be  difficult  or  impossible  to  have  completely  developed  a  team 
consensus  on  what  objective  evidence  must  be  seen  (in  advance).  But  sooner  or  later  the  ap¬ 
praisal  team  must  establish  a  consensus  view  on  what  is  reasonable  to  expect,  since  it  is  only 
the  presence  of  that  consensus  view  that  permits  a  determination  of  practice  implementation 
to  be  made. 

The  final  practice  implementation  determination  task  is  that  of  developing  a  team  consensus 
on  whether  the  practice  is  implemented  for  the  process  instantiation  being  examined.  This 
decision  is  based  on  the  difference  between  what  is  expected  and  what  is  observed. 
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Appendix  C  Focused  Investigation 

Elaboration  and  Guidance 


Concept  Description 

This  appendix  describes  the  use  of  preliminary  objective  evidence  review,  continuous  con¬ 
solidation  of  objective  evidence,  and  practice  characterization  in  focusing  the  data  collection 
and  investigation  effort  of  the  appraisal  team. 

Focused  investigation  relies  on  a  high  degree  of  planning,  organization,  and  subsequent  man¬ 
agement  and  control  of  the  activities  of  the  appraisal  The  concept  incorporates  the  following 
activities: 

•  collecting  preliminary  objective  evidence  through  instruments  as  a  part  of  obtaining  pre¬ 
liminary  data 

•  creating  an  inventory  of  objective  evidence  collected,  to  support  practice  implementation 
characterization 

•  initially  reviewing  and  analyzing  preliminary  objective  evidence  inventoried,  to  identify 
gaps  in  objective  evidence  supporting  practice  characterization 

•  identifying  information  needs  to  support  initial  preparation  and  refinement  of  the  data 
collection  plan 

•  continuously  consolidating  objective  evidence  collected  and  updating  the  status  of  prac¬ 
tice  characterization  for  each  organizational  unit  instantiation  (aggregated  up  to  the  or¬ 
ganizational  unit) 

Preliminary  Focused  investigation 

Focused  investigation  should  be  begun  in  the  Appraisal  Planning  phase  of  the  appraisal.  Fo¬ 
cused  investigation  is  best  initiated  with  a  practice-based  initial  data  collection  instrument 
that  documents  the  organizational  unit’s  implementation  of  the  practices  of  the  CMMI  model 
for  each  instantiation  within  the  scope  of  the  appraisal.  Preliminary  data  may  be  collected 
using  instruments  such  as  questionnaires,  surveys,  and  presentations.  This  data  should  in¬ 
clude  a  preliminary  inventory  of  Practice  Implementation  Indicators. 

An  inventory  and  review  of  this  data  provides  an  important  initial  determination  of  the  gaps 
in  the  data  available  supporting  practice  implementation,  as  well  as  what  information  and 
objective  evidence  is  needed.  These  activities  are  performed  as  part  of  the  Obtain  and  Ana- 
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lyze  Preliminary  Objective  Evidence  process.  The  more  complete  and  comprehensive  this 
early  data  collection  and  analysis  is,  the  better  prepared  the  organizational  unit  will  be  for  the 
appraisal. 

These  preliminary  information  needs  can  provide  the  foundation  of  the  data  collection  plan 
for  the  remainder  of  the  appraisal  process.  They  also  provide  the  foundation  for  the  readiness 
review  and  any  necessary  adjustments  in  the  appraisal  plan,  providing  a  clearer  set  of  initial 
expectations  for  the  magnitude  of  the  appraisal  effort.  Data  gaps  found  can  result  in  addi¬ 
tional  document  requests  and  other  data  collection  plans.  These  activities  are  performed  as  an 
early  part  of  the  Prepare  for  Collection  of  Objective  Evidence  process. 

Continuous  Consolidation  and  Tracking 

Following  the  initial  focused  investigation  effort,  a  data  collection  plan  is  developed  and  fol¬ 
lowed.  Data  collection  activities  are  described  by  the  Examine  Objective  Evidence  process. 
This  process  typically  consists  of  planned  data  collection  activities  that  include  presentations, 
document  reviews,  and  interviews. 


As  these  data  collection  activities  are  performed,  practice  characterization  and  strengths  and 
weaknesses  are  recorded  and  added  to  the  existing  objective  evidence  inventory,  and  continu¬ 
ously  reviewed  (see  the  Verify  and  Validate  Objective  Evidence  and  Document  Objective 
Evidence  processes).  Additional  data  collected  is  added  and  consolidated  with  the  data  al¬ 
ready  collected  to  continuously  provide  the  assessment  team  with  a  view  of  their  progress 
against  the  data  collection  plan  and  model  coverage.  This  is  referred  to  as  “continuous  con¬ 
solidation.” 


Monitoring  and  controlling  the  data  collection  plan  and  model  coverage  is  an  essential  aspect  of 
performing  focused  investigation  and  continuous  consolidation.  The  appraisal  team  must  be 
able  to  record,  monitor,  and  track  progress  against  the  data  collection  plan.  This  may  be  done  in 
several  ways,  but  generally  requires  the  use  of  some  mechanism  for  recording  the  progress  to¬ 
wards  determining  practice  characterization  for  each  reference  model  practice  within  the  scope 
of  the  appraisal.  As  data  is  collected  for  each  practice,  and  for  each  sample  instantiation  of  the 
organizational  unit  being  appraised,  it  is  also  useful  to  have  some  mechanism  for  easily 
comparing  and  consolidating  practice  implementation.  Instruments  and  automated  tools  that 
support  the  Conduct  Appraisal  phase  of  the  appraisal  can  greatly  facilitate  this. 

Perhaps  the  most  important  feature  of  focused  investigation  is  the  appraisal  team’s  awareness 
of  its  status  regarding  determination  of  practice  characterization  and  goal  satisfaction.  The 
team  continually  maintains  an  understanding  of  how  the  data  collected  supports  the  imple¬ 
mentation  of  each  practice  for  each  instantiation  of  the  organizational  unit,  and  what  addi¬ 
tional  objective  evidence  is  needed.  This  allows  the  team  to  update  the  data  collection  plan  to 
optimally  refocus  their  efforts  during  the  data  collection  activities. 
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'he  method  shall  be  documented  and,  at  a  minimum,  include  MDD,  VI. 


determining  the  appropriate  size  of  the  appraisal  team.  Team 

(ABC)  The  method  documentation  shall  provide  guidance  on  1.3  Select  and  Prepare  1.3.2  Select  Team  Members 
the  roles  and  responsibilities  of  appraisal  team  members.  Team 
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ARC  ID  I  ARC^quirement _ MDD  Process _ MDD  Activity _ Notes/Comments 

1.3. 1  .a  (ABC)  the  purpose  of  the  appraisal  1 .4  Obtain  and  Analyze  1 .4. 1  Prepare  Participants 
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ARC  ID _ ARC  Requirement _ MDD  Process _ MDD  Activity _ Notes/Comments _ 

i.5.2.a  (ABC)  The  observation  was  derived  from  objective  evidence  2.3  Document  Objective  2.3.3  Document  Practice  See  also  descriptions  of  verifying  practice 


indicator  types  (direct,  indirect,  affirmation) 
as  described  in  2.2.1. 
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■.6.3.a  (A)  Rate  the  goal  “satisfied”  when  the  associated  generic  or  2.4  Generate  Appraisal  2.4.1  Derive  Findings  and 

specific  practices  (or  acceptable  alternative  practices)  are  Results  Rate  Goals 

judged  to  be  implemented  and  the  aggregate  of  weaknesses 
does  not  have  a  significant  negative  impact  on  goal  achieve¬ 
ment. 
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Glossary 


The  MDD  glossary  defines  many,  but  not  all,  terms  used  in  this  document.  The  following  additional 

sources  for  terms  and  definitions  should  be  considered  supplementary  to  the  MDD  glossary: 

•  CMMI  model  glossary  and  terminology 

•  ARC  glossary 

Terms  that  are  particularly  significant  to  this  document  are  duplicated  from  the  model  docu¬ 
ment  or  ARC  for  convenience. 

accurate  An  observation  extracted  from  data  collected  during  an  appraisal 

observation  that  has  been  determined  by  the  appraisal  team  to  be  (a)  worded 

appropriately,  (b)  based  on  information  seen  or  heard,  (c)  relevant 
to  the  appraisal  reference  model  being  used,  (d)  significant  such 
that  it  can  be  classified  as  a  strength,  weakness,  or  alternative  prac¬ 
tice,  and  (e)  not  redundant  with  other  observations,  [arc  vi.i] 

affirmation  An  oral  or  written  statement  confirming  or  supporting  implementa¬ 

tion  of  a  CMMI  model  specific  practice  or  generic  practice.  Affir¬ 
mations  are  usually  provided  by  the  implementers  of  the  practice 
and/or  internal  or  external  customers,  but  may  also  include  other 
stakeholders  (e.g.,  managers,  suppliers),  [derived  from  mod  method  over¬ 
view]  Interview  responses  are  examples  of  face-to-face  affirmations. 
Alternative  forms  of  affirmations  could  include  presentations  or 
demonstrations  of  a  tool  or  mechanism  as  it  relates  to  implementa¬ 
tion  of  a  CMMI  model  practice,  [derived  from  mod  PII  appendix  B] 

alternative  practice  A  practice  that  is  a  substitute  for  one  or  more  generic  or  specific 

practices  contained  in  the  CMMI  model  that  achieves  an  equiva¬ 
lent  effect  toward  satisfying  the  goal  associated  with  the  practices. 
Alternative  practices  are  not  necessarily  one-for-one  replacements 
for  the  generic  or  specific  practices,  [arc  vi.i  and  cmmi  model  glossary] 

appraisal  An  examination  of  one  or  more  processes  by  a  trained  team  of  pro¬ 

fessionals  using  an  appraisal  reference  model  as  the  basis  for  de¬ 
termining,  as  a  minimum,  strengths  and  weaknesses,  [arc  vi.i] 
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Appraisal 
Disclosure 
Statement  (ADS) 


appraisal  findings 


appraisal  input 


appraisal  method 
class 


appraisal  modes  of 
usage 


appraisal 

objectives 

appraisal  output 


appraisal 

participants 

appraisal 

rating 


appraisal  record 


A  summary  statement  describing  the  ratings  generated  as  outputs 
of  the  appraisal,  and  the  conditions  and  constraints  under  which 
the  appraisal  was  performed.  The  ADS  should  be  used  for  public 
disclosures  of  maturity  level  or  capability  level  ratings  so  they  can 
be  interpreted  accurately,  [local] 


The  results  of  an  appraisal  that  identify  the  most  important  issues, 
problems,  or  opportunities  for  process  improvement  within  the 
appraisal  scope.  Appraisal  findings  are  inferences  drawn  from 
valid  observations.  [CMMI  model  glossary  and  ARC  vl.l] 

The  collection  of  appraisal  information  required  before  data 
collection  can  commence,  [iso  98C  and  arc  vi.i] 

A  family  of  appraisal  methods  that  satisfy  a  defined  subset  of  re¬ 
quirements  in  the  Appraisal  Requirements  for  CMMI  (ARC).  These 
classes  are  defined  so  as  to  align  with  typical  usage  modes  of  ap¬ 
praisal  methods.  [derivedfromARC  vI.O.CMMI  model  glossary  and  ARCvI.I] 

The  contexts  in  which  an  appraisal  method  might  be  utilized.  Ap¬ 
praisal  modes  of  usage  identified  for  the  SCAMPI  method  include 
internal  process  improvement,  supplier  selection,  and  process 
monitoring. 

The  desired  outcome(s)  of  an  appraisal  process,  [arc  vi.i] 


All  of  the  tangible  results  from  an  appraisal  (see  “appraisal  re¬ 
cord”).  [ISO  98C  and  ARC  vl.l] 

Members  of  the  organizational  unit  who  participate  in  providing 
information  during  the  appraisal.  [Cmmi  model  glossary  and  arc  vi.i] 

The  value  assigned  by  an  appraisal  team  to  either  (a)  a  CMMI  goal 
or  process  area,  (b)  the  capability  level  of  a  process  area,  or  (c)  the 
maturity  level  of  an  organizational  unit.  The  rating  is  determined 
by  enacting  the  defined  rating  process  for  the  appraisal  method 
being  employed.  [CMMI  model  glossary  and  arc  vl.l] 

An  orderly,  documented  collection  of  information  that  is  pertinent 
to  the  appraisal  and  adds  to  the  understanding  and  verification  of 


111-34 


CMU/SEI-2001-HB-001 


appraisal 
reference  model 

appraisal  scope 

appraisal  sponsor 


appraisal  tailoring 


appraisal 
team  leader 


artifact 


assessment 


capability 

evaluation 


consensus 


the  appraisal  findings  and  ratings  generated,  [derived  from  iso  98C  and 

ARCvl.l] 

The  CMMI  model  to  which  an  appraisal  team  correlates  imple¬ 
mented  process  activities.  [CMMI  model  glossary  and  ARC  vl.l] 

The  definition  of  the  boundaries  of  the  appraisal  encompassing  the 
organizational  limits  and  the  CMMI  model  limits  within  which  the 
processes  to  be  investigated  operate,  [derived  from  cmmi  model  glossary, 
ISO  98C  and  ARCvl.l] 

The  individual,  internal  or  external  to  the  organization  being  appraised, 
who  requires  the  appraisal  to  be  performed,  and  provides  financial  or 
other  resources  to  cany  it  out.  [derived  from  iso  98C  and  ARC  vl  .1] 

Selection  of  options  within  the  appraisal  method  for  use  in  a  spe¬ 
cific  instance.  The  intent  of  tailoring  is  to  assist  an  organization  in 
aligning  application  of  the  method  with  its  business  needs  and  ob- 

jeCtiveS.  [CMMI  model  glossary  and  ARC  vl.l] 

The  person  who  leads  the  activities  of  an  appraisal  and  has  satis¬ 
fied  the  qualification  criteria  for  experience,  knowledge,  and  skills 
defined  by  the  appraisal  method.  [ARC  vl.l] 

A  tangible  form  of  objective  evidence  indicative  of  work  being  per¬ 
formed  that  is  a  direct  or  indirect  result  of  implementing  a  CMMI 
model  practice.  (See  “direct  artifact”  and  “indirect  artifact.”) 

An  appraisal  that  an  organization  does  to  and  for  itself  for  the  pur¬ 
poses  of  process  improvement.  [ARC  vl.l] 

An  appraisal  by  a  trained  team  of  professionals  used  as  a  discrimi¬ 
nator  to  select  suppliers,  for  contract  monitoring,  or  for  incentives. 
Evaluations  are  used  to  gain  insight  into  the  process  capability  of  a 
supplier  organization  and  are  intended  to  help  decision  makers 
make  better  acquisition  decisions,  improve  subcontractor  perform¬ 
ance,  and  provide  insight  to  a  purchasing  organization.  [ARC  vi.i] 

A  method  of  decision  making  that  allows  team  members  to  de¬ 
velop  a  common  basis  of  understanding  and  develop  general 
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consolidation 


corroboration 


coverage 


coverage  criteria 


data  coliection 
session 


direct  artifact 


discovery-based 

appraisai 


agreement  concerning  a  decision  that  all  team  members  are  willing 
to  support.  [ARC  vl.l] 


The  activity  of  collecting  and  summarizing  the  information  pro¬ 
vided  into  a  manageable  set  of  data  to  (a)  determine  the  extent  to 
which  the  data  are  corroborated  and  cover  the  areas  being  investi¬ 
gated,  (b)  determine  the  data’s  sufficiency  for  making  judgments, 
and  (c)  revise  the  data-gathering  plan  as  necessary  to  achieve  this 
sufficiency.  [arCvI.i] 


The  extent  to  which  enough  data  has  been  gathered  to  confirm  that 
an  observation  is  acceptable  for  use  by  an  appraisal  team,  [arc  vi.i] 
In  SCAMPI,  corroboration  is  obtained  through  method  require¬ 
ments  for  the  collection  of  practice  implementation  indicators  of 
multiple  types  (see  “practice  implementation  indicator”). 

The  extent  to  which  objective  evidence  gathered  addresses  a  model 
component  within  the  scope  of  an  appraisal,  [arc  vi.i] 

The  specific  criterion  that  must  be  satisfied  in  order  for  coverage 
to  be  claimed,  [arc  vi.i] 

An  activity  during  which  information  that  will  later  be  used  as  the 
basis  for  observation  formulation  or  corroboration  is  gathered. 

Data  collection  sessions  (or  activities)  include  the  administration 
and/or  analysis  of  instruments,  document  review,  interviews,  and 
presentations.  [ARC  vl.l] 

The  tangible  outputs  resulting  directly  from  implementation  of  a 
specific  or  generic  practice.  An  integral  part  of  verifying  practice 
implementation.  May  be  explicitly  stated  or  implied  by  the  practice 
statement  or  associated  informative  material.  [MDD  method  overview] 


An  appraisal  in  which  limited  objective  evidence  is  provided  by 
the  appraised  organization  prior  to  the  appraisal,  and  the  appraisal 
team  must  probe  and  uncover  a  majority  of  the  objective  evidence 
necessary  to  obtain  sufficient  coverage  of  CMMI  model  practices. 
Discovery-based  appraisals  typically  involve  substantially  greater 
appraisal  team  effort  than  verification-based  appraisals,  in  which 
much  of  the  objective  evidence  is  provided  by  the  appraised  or- 


111-36 


CMU/SEI-2001-HB-001 


document 


findings 


focused 

investigation 


fully  implemented 
(FI) 


indirect  artifact 


instantiation 


ganization.  (See  verification-based  appraisal  for  contrast.) 

A  collection  of  data,  regardless  of  the  medium  on  which  it  is  re¬ 
corded,  that  generally  has  permanence  and  can  be  read  by  humans 
or  machines,  [arc  vi.i]  In  SCAMPI,  documents  are  work  products 
reflecting  the  implementation  of  one  or  more  model  practices.  This 
typically  includes  work  products  such  as  organizational  policies, 
procedures,  and  implementation-level  work  products.  Documents 
may  be  available  in  hardcopy,  softcopy,  or  accessible  via  hyper¬ 
links  in  a  web-based  environment,  [derived  from  MDD  method  overview] 

The  conclusions  of  an  assessment,  evaluation,  audit,  or  review  that 
identify  the  most  important  issues,  problems,  or  opportunities 
within  the  appraisal  scope.  Findings  include,  at  a  minimum, 
strengths  and  weaknesses  based  on  valid  observations,  [arc  vi.i] 

A  technique  to  prioritize  appraisal  team  effort  based  on  the  con¬ 
tinuous  collection  and  consolidation  of  appraisal  data,  and  moni¬ 
toring  of  progress  toward  achieving  sufficient  coverage  of  CMMI 
model  practices.  Appraisal  resources  are  targeted  toward  those  ar¬ 
eas  for  which  further  investigation  is  needed  to  collect  additional 
data  or  verify  the  collected  set  of  objective  evidence,  [derived  from 

MDD  method  overview] 


A  practice  characterization  value  assigned  to  a  process  instantia¬ 
tion  when  (1)  direct  artifacts  are  present  and  judged  to  be  appro¬ 
priate,  (2)  at  least  one  indirect  artifact  and/or  affirmation  exists  to 
confirm  the  implementation,  and  (3)  no  substantial  weaknesses  are 
noted.  [MDD  3.7.2] 

An  artifact  that  is  a  consequence  of  performing  a  specific  or  ge¬ 
neric  practice  or  that  substantiate  its  implementation,  but  which  is 
not  the  purpose  for  which  the  practice  is  performed.  This  indicator 
type  is  especially  useful  when  there  may  be  doubts  about  whether 
the  intent  of  the  practice  has  been  met  (e.g.,  a  work  product  exists 
but  there  is  no  indication  of  where  it  came  from,  who  worked  to 
develop  it,  or  how  it  is  used).  [MDD  method  overview] 

For  practices  implemented  by  projects,  each  project;  for  practices 
implemented  organization-wide,  the  instance. 
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instruments 


internal  process 
improvement  (IPI) 


interviews 


largely  implemented 
(LI) 


lead  appraiser 

mini-team 

not  implemented 
(Nl) 


Artifacts  used  in  an  appraisal  for  the  collection  and  presentation  of 
data  (e.g.,  questionnaires,  organizational  unit  information  packets). 
[ARC  vi.i]  In  SCAMPI,  instmments  are  used  to  collect  written  in¬ 
formation  relative  to  the  organizational  unit’s  implementation  of 
CMMI  model  practices.  This  can  include  assets  such  as  question¬ 
naires,  surveys,  or  an  organizational  mapping  of  CMMI  model 
practices  to  its  corresponding  processes. 


An  appraisal  mode  of  usage  in  which  organizations  appraise  inter¬ 
nal  processes,  generally  to  either  baseline  their  process  capability, 
to  establish  or  update  a  process  improvement  program,  or  to  meas¬ 
ure  progress  in  implementing  such  a  program,  [derived  from  mod 

method  overview] 


A  meeting  of  appraisal  team  members  with  appraisal  participants 
for  the  purpose  of  gathering  information  relative  to  work  processes 
in  place,  [arc  vi.i]  In  SCAMPI,  this  includes  face-to-face  interac¬ 
tion  with  those  implementing  or  using  the  processes  within  the 
organizational  unit.  Interviews  are  typically  held  with  various 
groups  or  individuals,  such  as  project  leaders,  managers,  and  prac¬ 
titioners.  A  combination  of  formal  and  informal  interviews  may  be 
held  and  interview  scripts  or  exploratory  questions  developed  to 
elicit  the  information  needed. 


A  practice  characterization  value  assigned  to  a  process  instantia¬ 
tion  when  (1)  direct  artifacts  are  present  and  judged  to  be  appro¬ 
priate,  (2)  at  least  one  indirect  artifact  and/or  affirmation  exists  to 
confirm  the  implementation,  and  (3)  one  or  more  weaknesses  are 
noted.  [MOD  3.7.2] 

A  person  who  has  achieved  recognition  from  an  authorizing  body 
to  perform  as  an  appraisal  team  leader  for  a  particular  appraisal 
method.  [ARC  vi.i] 


See  “process  area  mini-team.” 

A  practice  characterization  value  assigned  when  the  appraisal  team 
determines  insufficient  objective  evidence  exists  to  state  that  the 
practice  is  implemented.  That  is,  the  criteria  for  assigning  a  value 
of  Fully  Implemented  (FI),  Largely  Implemented  (LI),  or  Partially 
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Implemented  (PI)  are  not  satisfied,  [local] 


objective  evidence 


observation 


organizationai  unit 


partialiy 

implemented  (PI) 


practice 

characterization 


Qualitative  or  quantitative  information,  records,  or  statements  of 
fact  pertaining  to  the  characteristics  of  an  item  or  service  or  to  the 
existence  and  implementation  of  a  process  element,  which  is  based 
on  observation,  measurement,  or  test  and  which  can  be  verified. 
[CMMI  model  glossary,  ISO  98C  and  ARC  vl.l]  In  SCAMPI,  SOUrceS  of  objec¬ 
tive  evidence  include  instruments,  presentations,  documents,  and 
interviews. 

A  written  record  that  represents  the  appraisal  team  members’  un¬ 
derstanding  of  information  either  seen  or  heard  during  the  ap¬ 
praisal  data  collection  activities.  The  written  record  may  take  the 
form  of  a  statement  or  may  take  alternative  forms  as  long  as  the 
information  content  is  preserved.  [Cmmi  model  glossary ,  arc  vi.i] 

That  part  of  an  organization  that  is  the  subject  of  an  appraisal  (also 
known  as  the  organizational  scope  of  the  appraisal).  An  organiza¬ 
tional  unit  deploys  one  or  more  processes  that  have  a  coherent 
process  context  and  operates  within  a  coherent  set  of  business  ob¬ 
jectives.  An  organizational  unit  is  typically  part  of  a  larger  organi¬ 
zation,  although  in  a  small  organization,  the  organizational  unit 
may  be  the  whole  organization.  [Derived  ftom  cmmi  model  glossary,  ISO  98C 

and  ARC  vl  .l] 


A  practice  characterization  value  assigned  to  a  process  instantia¬ 
tion  when  (1)  direct  artifacts  are  absent  or  judged  to  be  inadequate, 
(2)  artifacts  or  affirmations  suggest  that  some  aspects  of  the  prac¬ 
tice  are  implemented,  and  (3)  weaknesses  have  been  documented. 
[MDD  3.7.2] 

The  assignment  of  a  value  describing  the  extent  to  which  a  CMMI 
model  practice  is  implemented,  used  as  a  mechanism  to  reach  ap¬ 
praisal  team  consensus.  The  range  of  values  for  practice  charac¬ 
terization  values  include  Fully  Implemented  (FI),  Largely  Imple¬ 
mented  (LI),  Partially  Implemented  (PI),  and  Not  Implemented 
(NI).  Practice  characterization  values  are  assigned  to  each  CMMI 
model  practice  for  each  process  instantiation  within  the  appraisal 
scope,  and  aggregated  to  the  organizational  unit  level,  [local] 
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practice 
implementation 
indicator  (Pll) 


preliminary  findings 


presentations 


process  area  mini¬ 
team 


process  context 


process  monitoring 


An  objective  attribute  or  characteristic  used  as  a  “footprint”  to  ver¬ 
ify  the  conduct  of  an  activity  or  implementation  of  a  CMMI  model 
specific  or  generic  practice.  Types  of  practice  implementation  in¬ 
dicators  include  direct  artifacts,  indirect  artifacts,  and  affirmations. 

[derived  from  15504-9  and  MDD  method  overview] 


Initial  findings  created  by  an  appraisal  team  after  consolidating 
and  synthesizing  valid  observations  to  provide  the  findings  to  ap¬ 
praisal  participants  for  validation  of  accuracy,  [derived  from  arc  vi  i] 

In  SCAMPI,  a  source  of  objective  evidence  that  includes  informa¬ 
tion  prepared  by  the  organization  and  delivered  visually  or  ver¬ 
bally  to  the  appraisal  team  to  aid  in  understanding  the  organiza¬ 
tional  processes  and  implementation  of  CMMI  model  practices. 
This  typically  includes  such  mechanisms  as  orientation  or  over¬ 
view  briefings,  and  demonstrations  of  tools  or  capabilities,  [derived 

from  MDD  method  overview] 


A  subset  of  the  appraisal  team  members,  typically  two  or  three, 
assigned  primary  responsibility  for  collection  of  sufficient  ap¬ 
praisal  data  to  ensure  coverage  of  their  assigned  reference  model 
process  areas,  [local] 


The  set  of  factors  documented  in  the  appraisal  input  that  influences 
the  judgment  and  comparability  of  appraisal  ratings.  These  include, 
but  are  not  limited  to,  (a)  the  size  of  the  organizational  unit  to  be 
appraised,  (b)  the  demographics  of  the  organizational  unit,  (c)  the 
application  domain  of  the  products  or  services,  (d)  the  size,  critical¬ 
ity,  and  complexity  of  the  products  or  services,  and  (e)  the  quality 
characteristics  of  the  products  or  services.  (Cmmi  model  glossary] 


An  appraisal  mode  of  usage  in  which  appraisals  are  used  to  moni¬ 
tor  process  implementation  (for  example,  after  contract  award  by 
serving  as  an  input  for  an  incentive/award  fee  decision  or  a  risk 
management  plan).  The  appraisal  results  are  used  to  help  the  spon¬ 
soring  organization  tailor  its  contract  or  process  monitoring  efforts 
by  allowing  it  to  prioritize  efforts  based  on  the  observed  strengths 
and  weaknesses  of  the  organization’s  processes.  This  usage  mode 
focuses  on  a  long-term  teaming  relationship  between  the  sponsor¬ 
ing  organization  and  the  development  organization  (buyer  and 
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supplier),  [derived  from  MDD  method  overview] 


process  profile 

rating 

satisfied 

strength 

sufficient  data 
coverage 

supplier  selection 

tailoring 

valid  observation 

verification-based 

appraisal 

weakness 


The  set  of  goal  ratings  assigned  to  the  process  areas  in  the  scope  of 
the  appraisal.  In  CMMI,  also  known  as  the  process  area  profile. 

[derived  from  IS098c  and  ARC  vl.l] 


(See  “appraisal  rating.”)  [CMMI  model  glossary  and  ARC  vl.l] 

Rating  given  to  a  goal  when  the  aggregate  of  valid  observations 
and  associated  findings  does  not  negatively  impact  achievement  of 
the  goal.  Rating  given  to  a  process  area  when  all  of  its  goals  are 
rated  “satisfied.”  [arc  vi.i] 

Exemplary  or  noteworthy  implementation  of  a  CMMI  model  prac¬ 
tice.  [CMMI  model  glossary  and  ARC  vl.l] 

A  determination  that  the  coverage  requirements  have  been  met. 

See  “coverage”  and  “coverage  criteria.”  [arc  vi.i] 

An  appraisal  mode  of  usage  in  which  appraisal  results  are  used  as 
a  high  value  discriminator  to  select  suppliers.  The  results  are  used 
in  characterizing  the  process-related  risk  of  awarding  a  contract  to 

a  supplier,  (derived  from  MDD  method  overview] 

See  “appraisal  tailoring.”  [arcvI.i] 

An  observation  that  the  appraisal  team  members  agree  is  (a)  accu¬ 
rate,  (b)  corroborated,  and  (c)  consistent  with  other  valid  observa¬ 
tions,  [arc  vi.i] 


An  appraisal  in  which  the  focus  of  the  appraisal  team  is  on  verify¬ 
ing  the  set  of  objective  evidence  provided  by  the  appraised  organi¬ 
zation  in  advance  of  the  appraisal,  in  order  to  reduce  the  amount  of 
probing  and  discovery  of  objective  evidence  during  the  appraisal 
on-site  period.  (See  discovery-based  appraisal  for  contrast.) 

The  ineffective,  or  lack  of,  implementation  of  one  or  more  CMMI 
model  practices.  [CMMI  model  glossary  and  ARC  vl.l] 
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